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SUMMARY  OF  THE  STUDY 

The  Chief  of  Naval  Education  and  Training  (CNET)  in  1975  directed  the 
establishment  at  local  training  activities  of  Curriculum  and  Instructional 
Standards  Offices  (CISOs).  These  offices  were  established  to  maintain  the 
quality  of  local  training  by  accomplishing  a  variety  of  training  development, 
instructional  support,  and  training  evaluation  functions.  Additional 
resources  were  not  provided  the  schools  to  implement  the  CISO  concept. 

Recent  concerns  over  the  ability  of  CISOs  to  perform  effectively  in 
their  assigned  roles  resulted  in  a  request  by  Chief  of  Naval  Technical  Train¬ 
ing  (CNTECHTRA)  Code  N63  for  a  study  of  the  CISO  concept.  Subsequently, 
the  Training  Analysis  and  Evaluation  Group  (TAEG)  was  tasked  by  CNET  to 
perform  the  study.  The  CNET  tasking  requested  TAEG  to:  (1)  evaluate  the 
CISO  concept  and  its  implementation  and  (2)  develop  options/recommendations, 
as  warranted,  for  accomplishing  required  training  development,  instructional 
support,  and  training  evaluation  functions. 

To  accomplish  the  study  objectives,  the  TAEG  collected  a  variety  of 
information  through  visits  to  and  questionnaires  sent  to  the  Naval  Education 
and  Training  Command  (NAVEDTRACOM)  CISOs.  Information  collected  concerned 
current  CISO  organizations  and  operations,  personnel  capabilities,  and  prob¬ 
lems  encountered.  Visits  to  Army  and  Air  Force  activities  were  also  made 
to  determine  how  these  services  accomplish  functions  comparable  to  those 
expected  of  Navy  CISOs. 

The  work  performed  led  to  the  conclusion  that  the  CISO  concept  is  basic¬ 
ally  sound.  A  number  of  positive  features  and  benefits  for  training  are 
associated  with  the  notion  of  an  office  within  a  local  training  activity 
that  is  specifically  charged  with  responsibility  for  assuring  training  quality. 
However,  the  CISO  concept  has  not  been  well  implemented  within  the  Command. 
Curriculum  and  Instructional  Standards  Offices  across  the  NAVEDTRACOM  are 
highly  variable  organizationally  and  in  terms  of  the  functions  they  perform. 

The  potential  effectiveness  of  these  offices  is  limited  by  a  lack  of  manpower 
(both  numbers  and  types  of  skills  required)  for  performing  quality  assurance 
functions  and  by  various  other  factors  that  affect  the  utilization  of  CISO 
personnel  at  the  local  training  activities. 

Two  sets  of  recommendations  for  overcoming  current  problems  and  improving 
CISO  contributions  to  training  quality  assurance  were  developed.  The  first 
set  concerns  the  total  CISO  concept.  These  recommendations,  presented  in 
section  VII  of  this  report,  are  provided  in  the  form  of  a  charter  for  CISO 
operations.  Specific  recommendations  concern  the  following  areas: 

•  functions  that  are  appropriate  for  CISO  staff  to  perform 

•  organization  and  staffing  of  CISOs 

•  interfaces  between  local  CISOs  and  CNTECHTRA 

•  mechanisms  for  ensuring  the  performance  of  training  quality  assur¬ 
ance  functions  at  local  command  levels. 
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The  second  set  of  recommendations,  presented  in  section  VIII,  concerns 
discrete,  separate  actions  that  could  be  taken  to  assist  generally  the  cause 
of  enhancing  the  quality  of  local  training.  They  are: 

1.  Develop  a  short  course  covering  purposes,  principles,  and  procedures 
of  Instructional  Systems  Development  (ISD)  and  require  that  at  least  all 
officers  reporting  to  training  command  billets  complete  the  course. 

2.  Develop  a  handbook  for  use  of  CISO  personnel  to  provide  guidance/ 
instructions  concerning  tasks  assigned  to  them.  This  handbook  should  contain 
information  such  as  how  to  conduct  task  analyses,  develop  tests,  requirements 
and  procedures  for  instructor  evaluation.  In  addition  to  providing  guidance, 
the  handbook  should  also  identify  sources  where  more  detailed  information 

on  specific  topics  can  be  found. 

3.  Develop  standardized  courses  in  training  quality  assurance  areas. 
These  courses  could  be  used  for  training  CISO  personnel.  They  could  also 

be  used  by  CISOs  for  inservice  training  of  other  locally  assigned  personnel. 

4.  Rewrite  instructions  governing  CISOs.  Currently,  several  CNET 
and  CNTECHTRA  instructions  assign  tasks  to  CISOs.  It  is  recommended  that 
these  taskings  be  appropriately  consolidated  and  that  , ne  instruction,  with 
cross  references  as  necessary,  be  written  and  promulgated  for  CISO  guidance. 

5.  Establish  a  direct  interface  between  CNTECHTRA  and  local  CISOs 
for  defined  purposes.  It  is  specifically  recommended  that  CNTECHTRA  N63  be 
appropriately  tasked,  staffed,  and  funded  to: 

•  maintain  direct  cognizance  over  local  CISO  operations 

•  assist  local  CISOs  in  resolving  problems  and  obtaining  resources 
or  training  needed  to  perform  assigned  work. 

6.  Establish  proficiency  billets  for  military  personnel  to  be  assigned 
to  CISOs. 
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SECTION  I 
INTRODUCTION 

Chief  of  Naval  Education  and  Training  (CNET)  Instruction  1540.6  (1975) 
required  the  establishment  at  CNET  training  activities  of  Curriculum  and 
Instructional  Standards  Offices  (CISOs).  These  offices  were  assigned  respon¬ 
sibility  for  accomplishing  a  variety  of  functions  related  to  promoting  the 
quality  of  training  given  at  local  levels.  These  functions  fall  generally 
within  three  broad  areas:  training  development,  instructional  support,  and 
training  evaluation.  Additional  resources  were  not  provided  to  the  schools 
for  compliance  with  the  instruction. 

Recently,  questions  have  arisen  at  CNET  and  CNET  Functional  Command 
levels  concerning  CISOs'  organizational  and  personnel  capabilities  for  per¬ 
forming  assigned  functions.  In  view  of  these  concerns,  CNET,  at  the  request 
of  the  Chief  of  Naval  Technical  Training  (CNTECHTRA),  Code  N63,  tasked  the 
TAEG  to  conduct  a  study  of  the  CISO  concept. 

PURPOSE 

The  purposes  of  this  study  were  to: 

•  evaluate  the  CISO  concept  and  its  implementation  within  the 
NAVEDTRACOM 

•  develop  options/recommendations  (as  warranted)  for  more  effective 
future  accomplishment  of  these  functions. 

ORGANIZATION  OF  THE  REPORT 

The  remainder  of  this  report  is  contained  in  seven  sections  and  four 
appendices.  Section  II  provides  detailed  information  about  the  CISO  concept. 
The  requirements  established  for  CISOs  by  various  higher  authority  instructions 
are  presented,  and  interfaces  between  local  schoolhouse  CISOs  and  other 
CNET  activities  are  described.  The  technical  approach  of  the  study  is  given 
in  section  III.  The  procedures  and  instruments  used  to  obtain  information 
about  current  CISO  operations  are  described.  Procedures  used  for  developing 
alternatives  to  the  CISO  concept  and  for  developing  recommendations  for  the 
future  operation  of  these  offices  are  also  described  in  section  III.  The 
results  of  the  CISO  evaluation  are  presented  and  discussed  in  sections  IV 
and  V.  Section  IV  presents  the  results  obtained  from  questionnaires.  Section 
V  presents  the  results  of  interviews  with  local  training  activity  personnel. 

A  number  of  alternatives  to  the  current  CISO  concept  as  a  means  of  assuring 
the  quality  of  Navy  training  are  presented  in  section  VI.  Section  VII  presents 
a  proposed  charter  for  CISO  operations.  The  final  section  of  the  report, 
section  VIII,  presents  recommendations. 

Appendix  A  lists  the  functions  that  CISOs  are  currently  expected  to 
perform.  A  copy  of  the  questionnaire  package  used  to  obtain  information 
about  CISO  operations  is  contained  in  appendix  B.  Information  concerning 
the  performance  by  the  Army  and  Air  Force  of  training  functions  comparable 
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to  those  expected  of  Navy  CISOs  is  presented  in  appendix  C.  Appendix  D 
contains  technical  notes  concerning  the  statistical  treatment  of  data 
obtained  from  the  questionnaires  used  in  the  study. 
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SECTION  II 

THE  CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE 
(CISO)  CONCEPT 

This  section  describes  the  Navy  CISO  concept.  Instructions  relevant 
to  CISOs  are  reviewed.  Interfaces  between  local  CISOs  and  other  NAVEDTRACOM 
activities  are  also  described. 

BASIC  INSTRUCTIONS 

Two  key  instructions  discussed  below  established  and  implemented  the 
CISO  concept  within  the  NAVEDTRACOM. 

CNET  INSTRUCTION  1540.6.  CNET  Instruction  1540.6  directed  the  establishment 
of  CISOs  at  training  activities.  This  organizational  concept  was  implemented 
to  perform  curriculum  development  and  appraisal  functions  more  effectively. 

The  CISO  concept  was  defined  as  a  subsystem  because  it  was  designed  to  func¬ 
tion  within  the  systems  approach  to  training.  The  instruction  provided 
that  the  CISOs  "...  will  function  as  an  integral  component  of  the  activity 
they  serve  and  not  as  an  extension  of  higher  authority." 

Curriculum  and  Instructional  Standards  Offices  were  charged  to  maintain 
quality  assurance  of  training  within  prescribed  standards  through  review  of 
curricula  and  documentation,  classroom  monitoring,  guidance,  inspection, 
maintenance  of  publications  and  training  aids,  and  instructor  and  staff 
inservice  training  programs.  They  were  also  assigned  responsibility  for 
maintaining  local  testing  programs  within  prescribed  standards  through  per¬ 
forming  functions  such  as  construction  and  validation  of  tests  and  analysis 
of  test  data.  Other  functions  assigned  included  collecting,  processing, 
evaluating,  and  reporting  feedback  data  regarding  training  quality  using 
questionnaires,  student  critiques,  structured  interviews. 

The  CNET  instruction  further  provided  that  the  CISOs  be  headed  by  a 
special  assistant  reporting  to  the  commanding  officer  of  the  training  activity. 
Curriculum  and  Instructional  Standards  Offices  were  to  be  administratively 
separate  from  the  instructional  departments  but  provide  them  technical  assist¬ 
ance  in  accomplishing  quality  training.  Two  branches— an  Evaluation  Branch 
and  a  Curriculum  and  Training  Support  Branch--were  specified  for  a  typical 
CISO  organization. 

Evaluation  Branch  responsibilities  include  maintaining  item  banks  for 
preparing  examinations  for  use  by  the  instructional  departments,  statistical 
analyses  of  test  data  for  use  in  training  evaluation  and  test  item  improvement, 
internal  and  external  feedback,  developing  proposals  for  short-and  long¬ 
term  evaluation  projects,  and  student  critique  programs.  Responsibilities 
of  the  Curriculum  and  Training  Support  Branch  include  task  analysis,  curriculum 
development,  monitoring  classroom  instruction,  instructor  in-service  training, 
monitoring  procurement  of  training  devices,  developing  training  aids,  and 
maintaining  a  central  technical  library.  The  CNET  instruction  (1540.6) 
which  established  the  CISO  concept  at  the  local  levels,  however,  specifically 
noted  that  additional  manpower  would  not  be  assigned  to  training  activities 
to  implement  the  concept. 
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CNTECHTRA  INSTRUCTION  1540.40.  Chief  of  Naval  Technical  Training  Instruc¬ 
tion  1540.40  (1979)1  formally  implemented  the  CISO  concept  within  CNTECHTRA 
training  activities.  This  instruction  is  also  used  for  guidance  by  Commander, 
Training  Command,  U.S.  Atlantic  Fleet  (COMTRALANT)  and  Commander,  Training 
Command,  U.S.  Pacific  Fleet  (COMTRAPAC)  activities. 

CNTECHTRA  Instruction  1540.40  prescribes  the  organization,  responsi¬ 
bilities,  and  duties  of  CISOs.  Functions  assigned  are  essentially  the  same 
as  those  identified  in  the  parent  CNET  instruction  (1540. 6).  Three  represen¬ 
tative  CISO  organizations  are  described  in  CNTECHTRA  Instruction  1540.40  and 
traininq  activities  are  identified  for  which  one  of  the  three  organizations 
was  considered  appropriate.  Types  of  personnel  considered  appropriate  to 
perform  the  various  CISO  functions  were  also  identified. 

OTHER  INSTRUCTIONS 

Other  CNET  instructions  also  place  requirements  on  local  CISOs.  These 
are  reviewed  briefly  below. 

CNET  INSTRUCTION  1550.15.  CNET  Instruction  1550.15  (1980)  defines  policy 
and  establishes  general  guidance  relating  to  Instructional  Program  Develop¬ 
ment  Centers  (IPDCs).  The  instruction  defines  an  Instructional  Program 
Manager  (IPM)  as  "the  agent  authorized  and  responsible  for  the  execution  of 
a  developed  instructional  program."  It  provides  (paragraph  4a(2))  opportunity 
for  the  CISO,  as  an  extension  of  the  IPM,  to  become  actively  involved  in 
the  instructional  program  development  (IPD)  process  in  a  number  of  "minimum 
areas."  These  include: 

•  participating  in  developing  and  implementing  validation  and 
evaluation  plans  and  processes 

•  acquiring  and  maintaining  statistical  data  for  IPD-developed 
courses 

•  providing  inputs  to  and/or  participating  in  preliminary  IPD 
Project  Plan  Workshops 

•  functioning  in  a  consultant  and/or  advisory  role  throughout  the 
entire  IPO  process 

•  identifying  and  documenting  revision  requirements  after 
implementing  an  IPD-developed  course. 

CNET  INSTRUCTION  1550. 15A.  CNET  Instruction  1550. 15A  (draft),  if  approved, 
will  cancel  CNET  Instruction  1550.15.  The  draft  instruction  (1550. 15A) 
describes  procedures  to  be  used  for  curriculum  development  projects 
assigned  to  IPDCs.  Paragraph  3c  of  the  draft  instruction  provides  for  the 
establishment  of  an  IPD  project  team  to  guide  each  assigned  project  through 
l 
i 

*CIS0  functions  were  previously  prescribed  in  an  enclosure  to  CNTECHTRA 
Instruction  5453. IB. 
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the  first  three  ISD  phases  of  analysis,  design,  and  development.  Members 
of  the  IPO  project  team  will  include  a  "...Curriculum  and  Instructional 
Standards  Office  (CISO)  representative  (or  equivalent)  and  a  representative 
of  the  implementing  school(s)."  The  responsibilities  of  the  CISO  represen¬ 
tative  (paragraph  4b (1(b))  will  be  to: 

•  provide  data,  information,  and  materials  for  existing  training  as 
required 

•  participate  in  developing/conducting  validation  and  evaluation 
plans. 

CNET  INSTRUCTION  1550.18.  CNET  Instruction  1550. IB  (draft)  also  has  indirect 
implications  for  local  CISO  duties  and  responsibilities.  This  draft  instruc¬ 
tion  assigns  responsibilities  to  the  CNET  Functional  Commands  for  applying 
ISD  procedures  to  selected  classes  of  courses.  Requirements  for  CISOs  are 
not  directly  stated.  However,  consonant  with  the  requirements  established 
for  CISOs  by  the  basic  CNET  and  CNTECHTRA  instructions,  these  offices  will 
likely  be  heavily  involved  in  applying  ISD  procedures  to  local  training. 

CNET  INSTRUCTION  1540.38.  CNET  Instruction  1540. 3B  (1981)  has  implications 
for  local  CISO  functions,  duties,  and,  especially,  staffing.  The  instruction 
requires  that  Functional  Commanders  ensure  that  the  capability  for  evaluating 
and  interpreting  training  appraisal  system  (TAS)  findings  exists  at  each 
training  activity.  Normally,  this  capability  would  be  resident  within  a 
local  CISO.  Accordingly,  compliance  with  this  instruction  has  implications 
for  the  type  of  talent  that  should  be  available  within  that  office. 

CISO  INTERFACES  WITH  HIGHER  AUTHORITY 

The  basic  instructions  governing  CISOs  assert  that  these  offices  are 
an  integral  part  of  the  activity  which  they  serve  and  not  agents  of  higher 
authority.  Consequently,  direct  tasking  of  a  CISO  is  the  prerogative  of 
the  local  commanding  officer  only  since  he  has  overall  responsibility  for 
the  quality  of  training  given  at  his  command.  However,  the  importance  to 
training  quality  of  the  functions  assigned  to  CISOs  cannot  be  denied.  Conse¬ 
quently,  the  activities  and  achievements  of  these  offices  are  of  interest 
to  a  number  of  higher  command  agencies.  Interfaces  between  local  CISOs  and 
higher  levels  of  the  NAVEDTRACOM  are  discussed  below.  These  interfaces 
are  manifest  in  various  ways;  e.g.,  vested  interests  in  high  quality  training, 
commonality  of  functions  performed  at  the  different  command  levels,  and 
provision  of  advice /support  to  local  CISOs. 

CNTECHTRA  INTERFACES.  CISO  activities  and  achievements  are  of  most  direct 
interest  to  CNTECHTRA  N63  and  to  CNTECHTRA  Training  Program  Coordinators 
(TPCs). 

CNTECHTRA  N63.  The  most  direct  link  between  local  CISOs  and  CNTECHTRA  staff 
is  with  the  CNTECHTRA  Instructional  Systems  Development,  Training  Standards 
and  Academic  Liaison  Branch  (N63).  The  N63  code  is  assigned  broad  respon¬ 
sibilities  for  developing,  supervising,  and  coordinating  of  training  programs. 
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particularly  with  respect  to  curriculum  and  instructional  standards  and 
evaluation.  The  duties  and  responsibilities  of  N63  (formerly  016)  as  related 
to  CISOs  are  set  forth  in  CNTECHTRA  Staff  Instruction  5400. 2D.  They 
include: 

•  planning,  implementing,  and  managing  the  application  of 
curriculum  and  instructional  standards  within  the  Technical 
Training  Command  (TECHTRACOM) 

•  planning,  developing,  and  coordinating  the  qualitative  evaluation 
requirements  of  technical  training  programs  and  courses 

•  planning,  coordinating,  and  managing  the  process  of  instructional 
systems  development  within  the  TECHTRACOM  and  directing  its 
application  to  current  and  future  training  programs  and  courses. 

Thus,  N63  is  concerned  with  many  of  the  same  functions  that  CISOs 
perform  at  local  training  activities.  N63  is  also  designated  formally  as 
the  TECHTRACOM  interface  with  CISOs  within  the  command.  This  group  has 
recently  initiated  workshops  for  CISO  staffs  which  are  aimed  at  providing 
assistance  and  upgrading  skills  for  personnel  assigned  to  local  training 
activities.  Other  recent  initiatives  on  behalf  of  CISO  personnel  include 
preparing  and  disseminating  a  newsletter  to  keep  CISO  staffs  informed  on 
matters  that  are  relevant  to  their  interests. 

Training  Program  Coordinators.  Training  Program  Coordinators  functioning 
within  several  staff  codes  at  CNTECHTRA  also  interface  with  local  CISO 
personnel.  The  nature  of  this  interface  is  not  formally  prescribed. 

However,  TPCs  exercise  general  supervision  at  command  level  over  matters 
pertaining  to  particular  courses.  The  usual  case  is  that  TPCs  work 
directly  with  the  local  school  staff  rather  than  with  the  CISO  personnel 
who  support  those  local  school  staffs.  However,  direct  contact  between 
TPCs  and  CISOs  may  occur  in  various  areas  such  as  for  curriculum  approval 
or  revision. 

CNET  INTERFACES.  Interfaces  between  CISOs  and  CNET  staff  functions  exist 
indirectly  by  way  of  a  shared  interest  in  training  quality.  The  activities 
of  CISOs  are  of  interest  to  several  CNET  staff  codes  even  though  there  are 
no  formally  established  direct  links  between  CNET  and  local  CISOs.  CNET 
staff  functions  having  most  interest  in  CISOs  are  identified  below. 

CNET  00A2,  Special  Assistant  for  Education  and  Training  Audits  and 
Technical  Assistance.  The  CNET  Special  Assistant  for  Education  and 
Training  Audits  and  Technical  Assistance  serves  as  a  primary  liaison 
between  CNET  headquarters  and  field  activities.  The  duties  of  this  office 
include  obtaining  and  providing  information  to  the  CNET  and  his  Principal 
Civilian  Advisor  concerning  the  status  of  instruction  and  instructional 
support  at  the  school  level.  Interactions  with  local  CISOs  are  required 
for  this  purpose. 

CNET  015,  Special  Assistant  for  Training  Appraisal.  The  CNET  Special 
Assistant  for  Training  Appraisal  is  responsible  for  collecting  and 


14 


TAEG  Report  No.  110 


disseminating  training  feedback  information  obtained  from  sources  external 
to  the  NAVEDTRACOM.  Although  local  training  activities  may  no  longer  collect 
training  feedback  from  fleet  sources,  they  participate  in  the  command's  TAS 
program.  Specifically,  local  training  activities  prepare  task  statements 
for  use  on  NAVEDTRACOM  TAS  Level  II  questionnaires.  These  questionnaire 
items  are  based  on  learning  objectives  of  a  course  in  question.  CNET  015 
also  sends  local  training  activities  summary  data  reflecting  areas  where 
schoolhouse  training  was  judged  by  graduates'  fleet  supervisors  to  be  inade¬ 
quate.  Local  schoolhouses  are  required  to  interpret  the  meaning  of  observed 
def iciencies,  assess  the  action  implications  of  these  deficiencies,  and 
initiate  appropriate  corrective  action.  These  external  evaluation  tasks 
fall  within  the  responsibilities  assigned  to  CISOs. 

CNET  N2,  Recruit  and  Specialized  Training  Operations.  The  CNET  Assistant 
Chief  of  Staff  (ACOS)  for  Recruit  and  Specialized  Training  Operations  (N2) 
serves  as  the  primary  point  of  contact  for  CNET-sponsored  training.  Code 
N2  monitors  and  assures  the  quality  of  NAVEDTRACOM  training.  This  function 
is  accomplished  largely  through  interactions  with  appropriate  TPCs.  CISO 
operations  as  they  affect  training  quality  are  of  direct  interest  to  this 
group. 

CNET  N5,  Research,  Development,  Test  and  Evaluation.  The  activities  of 
CNET  N5  (Research,  Development,  Test  and  Evaluation)  are  not  strongly  related 
to  local  CISO  activities.  However,  this  code  is  concerned  with  instructional 
innovations  and  implementing  improved  training  techniques  within  the  NAVEDTRACOM. 
Disseminating  information  to  CISOs  who  are  responsible  for  maintaining  familiar¬ 
ity  with  instructional  technology  improvements  may  assist  CISOs  to  improve 
training  at  local  levels. 

CNET  N9,  Training  Systems  Management.  The  ACOS  for  Training  Systems  Manage- 
ment  ( N9)  advises  and  acts  for  CNET  on  all  matters  pertaining  to  acquiring, 
developing,  evaluating,  and  maintaining  training  systems  including  training 
devices,  training  equipment,  curriculum,  and  other  instructional  materials. 

The  N9  code  is  responsible  for  implementing  and/or  assuring  the  ISD  approach 
is  used  within  the  NAVEDTRACOM.  This  ACOS  may  interface  directly  with  local 
CISOs  in  developing  training  to  be  conducted  at  local  activities  (see  CNET 
Instruction  1550.15)  or  indirectly  when  courses  are  developed  using  ISD  at 
local  levels. 

Instructional  Program  Development  Centers  which  function  in  various 
locations  under  the  cognizance  of  CNET  N9  will  increasingly  interface  with 
local  CISO  staff  for  developing  training.  Two  way  interfaces  exist  with 
CISO  representatives  supporting  the  IPDCs  in  developing  courses  for  which 
the  IPDC  is  responsible  and,  conversely,  with  IPDCs  supporting  the  local 
activity  in  training  development  for  which  the  Functional  Commanders  have 
responsibility. 

SUMMARY  STATEMENT 

The  review  presented  above  indicates  clearly  that  much  is  expected  of 
local  CIS0s--off ices  which  were  created  within  the  framework  of  currently 
available  manpower.  How  well  CISOs  perform  their  duties  and  how  well  the 
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functions  identified  for  ensuring  a  high  level  of  training  quality  are 
performed  at  the  local  level  are  of  direct  concern  not  only  to  the 
NAVEDTRACOM  but  to  the  Navy  at  large.  Such  concerns  led  directly  to  the 
request  for  the  present  study  to  evaluate  the  CISO  concept  and  to  develop 
alternatives/recommendations  (as  needed)  for  establishing  and  maintaining 
the  quality  of  technical  training. 
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SECTION  III 
TECHNICAL  APPROACH 

This  section  presents  the  technical  approach  used  to  obtain  and  process 
information  for  evaluating  the  CISO  concept.  The  approach  used  to  develop 
recommendations/options  for  accomplishing  training  development,  instructional 
support,  training  evaluation,  and  related  functions  is  also  discussed.  Fre¬ 
quently,  throughout  this  report  these  functions  are  grouped  under  the  term, 
"Training  Quality  Assurance." 

EVALUATION  OF  THE  CISO  CONCEPT 

A  major  objective  of  the  study  involved  an  evaluation  of  the  CISO  concept 
and  its  current  implementation  within  the  NAVEDTRACOM.  To  satisfy  this 
objective,  a  variety  of  information  concerning  benefits  associated  with  the 
CISO  concept,  functions  performed  by  local  CISOs,  staff  capabilities,  and 
problems  encountered  by  these  offices  was  required.  Necessary  information 
was  obtained  for  the  most  part  through  a  survey  of  NAVEDTRACOM  CISOs. 

CISO  SURVEY.  As  a  first  step  in  the  study  program,  a  list  of  functions 
which  CISOs  are  currently  expected  to  perform  was  compiled  from  applicable 
CNET  and  CNTECHTRA  instructions.  The  list  was  reviewed  by  CNET  and  CNTECHTRA 
staff  at  orientation  conferences  conducted  by  TAEG  at  each  headquarters  at 
the  beginning  of  the  program.  Both  staffs  were  requested  to  suggest  addi¬ 
tional  functions  that  they  thought  CISOs  should  perform  as  well  as  delete 
any  they  felt  CISOs  should  not  do.  The  resulting  list  of  functions  is  pre¬ 
sented  in  appendix  A. 

A  subset  of  functions  was  selected  from  the  overall  list  and  used  as  a 
basis  for  surveying  current  CISO  operations  and  local  commands'  attitudes 
toward  the  CISO  concept.  Information  concerning  CISO  operations,  command 
attitudes  towards  training  quality  assurance,  and  CISO  capabilities  for 
performing  the  functions  was  obtained  through  questionnaires  and  visits  to 
training  activities. 

Questionnaires.  Two  different  questionnaire  forms  were  developed  for  collect¬ 
ing  data  concerning  CISOs: 

1.  a  short  form  which  was  used  to  obtain  information  from  commanding 
officers/executive  officers,  CIS  officers,  and  training  department  heads 

2.  a  long  form  which  was  used  with  all  personnel  currently  filling 
CISO  billets/positions  at  the  local  commands. 

The  short  form  consisted  of  three  sections.  The  first  section  asked 
the  respondents  to  rank  11  general  training  support  functions  in  terms  of 
importance  to  training  quality.  The  second  section  asked  respondents  to 
give  their  preference  concerning  where  primary  responsibility  for  accom¬ 
plishing  these  functions  should  lie.  The  third  section  asked  for  ratings 
of  the  CISO  concept  and  of  the  performance  of  the  local  CISO. 
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The  long  form  consisted  of  five  sections.  The  first  asked  for  informa¬ 
tion  about  the  respondents;  e.g.,  position,  grade,  rate  or  rank,  time  in 
position,  and  educational  background.  The  second  section  addressed  the 
order  of  importance  of  increases  in  virious  resources  available  to  CISOs, 
and  the  usefulness  of  short  courses  o'  seminars  in  various  topics  related 
to  training  quality  assurance.  The  tiird  section  dealt  with  the  frequency 
and  content  of  communications,  both  within  the  CISO  and  between  CISO  personnel 
and  personnel  in  other  activities.  Sections  IV  and  V  listed  24  functions 
chosen  to  represent  all  the  functions  on  the  longer  list  that  CISOs  are 
expected  to  perform.  Questions  asked  concerned  the  degree  of  involvement 
with  the  functions,  frequency  of  performance,  and  amount  of  effort  expended. 
Questions  were  also  asked  concerning  ;he  impact  on  training  quality  of  elim¬ 
inating  those  functions. 

Appendix  B  contains  a  sample  questionnaire  package  consisting  of  a 
transmittal  letter  and  botn  short  and  long  forms.  These  packages  were  either 
transmitted  by  mail  or  band-carried  to  activities  visited.  Questionnaire 
packages  were  distribu{f-J  to  3n  NAVEDTRACOM  CISOs. 

Visits.  In  addition  to  ;  c  information  obtained  via  questionnaires,  a  variety 
of  other  informatior  concerning  CISO  organizations  and  operations  was  obtained 
through  visits  to  selected  training  activities.  Twenty  CISOs  were  visited. 
Interviews  were  conducted  with  local  CIS  officers,  key  CISO  staff,  and  various 
other  local  command  per-^nnel  (e.g.,  commanding  and  executive  officers, 
training  department  or  school  heads).  Questions  asked  concerned: 

•  formal  and  informal  CISO  group  structure 

•  interrelationships  with  other  local  departments  and  non-local 
agencies  and  commands 

•  tasking  lines 

•  authorized  manning  (numbers  and  types  of  skills) 

•  local  internal  and  external  evaluation  efforts/programs 

•  CISO-type  functions  performed  locally  by  individuals  not  assigned 

to  the  CISO 

•  resources  available  to  local  CISOs 

•  resources  needed  for  effective  functioning. 

In  addition,  candid  discussions  were  held  with  local  CISO  staffs  concern¬ 
ing  the  functions  they  actually  perform  and  factors  which  limit  their  ability 
to  perform  effectively.  Topics  discussed  included  the  local  command's  attitudes 
towards  training  quality  assurance,  resource  problems,  collateral  duties 
assigned  CISOs,  and  a  variety  of  other  related  matters.  Also,  discussions 
were  held  concerning  the  type  of  organization(s)  and  support  that  local 
CISO  staff  felt  would  be  most  helpful  in  performing  their  jobs.  Organization 
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charts  and  command  instructions  affecting  CISOs  were  also  collected  from 
the  local  activities. 

Data  Processing  and  Analysis.  For  processing  and  managing  necessary  data 
sets  and  for  statistical  analysis  of  data  obtained  from  the  questionnaires, 
the  Statistical  Package  for  the  Social  Sciences  (SPSS)  software  package 
(Nie,  Hull,  Jenkins,  Steinbrenner,  and  Bent,  1975;  Hull  and  Nie,  1979)  was 
used.  Information  obtained  from  visits,  local  instructions,  and  organiza¬ 
tional  charts  was  reduced  through  content  analysis. 

DEVELOPING  RECOMMENDATIONS/OPTIONS  FOR  QUALITY  ASSURANCE 

The  second  objective  of  the  study  concerned  developing  recommendations/ 
options  for  more  effective  future  accomplishment  of  training  quality  assurance 
functions.  Alternatives  to  the  CISO  concept  and  recommendations  for  CISO 
operations  were  developed  on  the  basis  of  TAEG  professional  judgment.  This 
judgment  was  aided  by  a  variety  of  inputs  from  various  sources.  These 
included: 


•  opinions  expressed  by  CNET  and  CNTECHTRA  staffs  and  local  CISO 
and  other  local  command  personnel  concerning  what  CISOs  should  or 
should  not  do  and  how  they  should  be  organized  and  staffed 

•  results  obtained  from  the  CISO  survey  concerning  current  CISO 
operations,  functions  performed,  personnel  capabilities,  training 
needs,  and  problems  that  should  be  overcome  to  promote  the  effective¬ 
ness  of  training  at  the  local  level 

•  practices  employed  by  the  Army  and  Air  Force  to  establish  and 
maintain  the  quality  of  their  training. 

Information  concerning  how  the  other  services  accomplish  functions 
similar  to  those  expected  of  NAVEDTRACOM  CISOs  was  obtained  from  visits 
made  to  Army  and  Air  Force  headquarters  and  several  field  units.  Informa¬ 
tion  obtained  from  interviews  with  other  services'  staffs  and  by  study  of 
printed  materials  provided  by  Army  and  Air  Force  activities  is  presented  in 
appendix  C.  The  practices  of  these  services  significantly  influenced  the 
thinking  of  the  TAEG  project  staff  with  respect  to  forming  recorrmendations 
for  future  Navy  quality  assurance  practices.  Consequently,  careful  reading 
of  this  appendix  is  recommended. 
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SECTION  IV 

RESULTS  AND  DISCUSSION  OF  CISO  EVALUATION: 

QUESTIONNAIRES  AND  ORGANIZATION  CHARTS 

This  section  presents  and  discusses  the  results  obtained  from  the  question¬ 
naire  portion  of  the  CISO  survey  and  from  the  analysis  of  CISO  organization 
charts.  A  summary  of  the  results  is  presented  first.  This  is  followed  by 
detailed  discussions.  Readers  not  interested  in  the  details  of  data  analysis 
may  obtain  the  basic  information  from  the  summary  and  proceed  directly  to 
section  V. 

SUMMARY  OF  QUESTIONNAIRE  FINDINGS 

A  nontechnical  summary  of  the  major  questionnaire  findings  is  presented 
below: 


1.  Commanding  or  executive  officers,  training  department  heads,  and 
CIS  officers  as  a  group  (n=134)  rated  11  general  training  functions  in  terms 
of  their  contribution  to  maintaining  training  quality.  The  six  most  important 
functions  were,  in  descending  order,  curriculum  development,  curriculum 
revision,  instructor  training,  internal  evaluation,  test  development  and 
revision,  and  instructor  evaluation. 

2.  The  same  respondents  clearly  prefer  that  the  six  most  important 
functions  be  done  locally  by  training  departments  and  CISOs,  not  by  outside 
activities  (e.g.,  IPOCs  or  higher  commands). 

3.  Both  commanding  or  executive  officers  and  CIS  officers  were  generally 
satisfied  with  the  CISO  concept  and  its  implementation.  However,  training 
department  heads  rated  CISOs  significantly  lower  than  commmanding  or  executive 
officers  and  CIS  officers.  Training  department  heads  were  not  entirely 
satisfied  with  the  CISO  concept  and  its  implementation. 

4.  There  are  five  groups  of  functions  with  which  CISO  personnel  report 
similar  degrees  of  involvement.  These  groups  of  functions  may  represent 
major  job  types.  Each  individual  tends  to  report  the  same  degree  of  involve¬ 
ment  for  all  functions  within  a  group.  The  first  of  these  five  groups  of 
functions  (CISWRK)  comprises  five  working  CIS  functions:  reviewing  documen¬ 
tation,  task  analysis,  curriculum  development,  course  design  and  revision, 
and  developing  training  aids.  The  second  group  (EVAL1)  includes  nine  evalua¬ 
tion  functions  concerning  student  testing,  internal  and  external  feedback, 
attrition,  setbacks,  and  student  critiques.  The  third  (CISDEV)  is  composed 
of  developmental  curriculum  and  instructional  support  functions  such  as 
monitoring  developments  in  training  technology,  selecting  instructional 
delivery  systems,  developing  and  conducting  inservice,  evaluating  inservice 
programs,  and  conducting  special  research  projects.  The  fourth  group  (EVAL2) 
consists  of  evaluating  instructors  and  conducting  annual  course  reviews. 

The  last  group  of  functions  (TECLIB)  consists  of  one  function,  that  of  main¬ 
taining  a  central  technical  library. 
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5.  The  mean  degree  of  involvement  with  three  groups  of  functions 
(CISDEV,  EVAL2,  and  TECLIB)  differs  significantly  among  CISOs.  This  is 
evidence  that  CISOs  actually  operate  differently  (i.e,  do  different  things) 
and  have  different  levels  of  involvement  in  functions. 

6.  There  are  six  clusters  (or  types)  of  CISOs  based  on  their  degree 
of  involvement  in  three  function  groups  (CISDEV,  EVAL2,  and  TECLIB).  The 
clusters  are  shown  in  table  6,  page  36.  CISOs  within  a  cluster  tend  to  do 
the  same  things  to  the  same  degree. 

7.  The  six  clusters  also  show  significant  differences  in  the  mean 
ratings  received  from  commanding  or  executive  officers,  CIS  officers,  and 
training  department  heads.  The  most  highly  rated  cluster  of  CISOs  shows  a 
high  degree  of  involvement  with  three  groups  of  functions,  CISDEV,  EVAL2, 
and  TECLIB  (see  4  on  previous  page).  Personnel  in  the  CISOs  in  this  cluster 
report  that  they  have  spent  more  time  attending  military  management  seminars. 
The  lowest  rated  cluster  of  CISOs  shows  less  involvement  with  CISDEV  and 
EVAL2  functions  and  shows  a  high  proportion  of  personnel  with  bachelor's 
degrees  and  a  higher  proportion  of  officers. 

8.  CISOs  vary  with  respect  to  the  number  of  functions  they  perform. 

The  number  of  functions  that  CISOs  report  no  involvement  with  ranges  from  0 
(i.e.,  they  are  involved  with  all  functions)  to  12  (i.e.,  they  only  do  half 
the  functions  listed).  The  number  of  functions  that  CISOs  are  involved 
with  has  no  relationship  to  the  ratings  received. 

9.  There  is  a  core  of  about  10.  functions  that  every  CISO  reports 
some  involvement  with.  These  10  include  reviewing  course  documentation, 
developing  curricula,  designing  courses,  developing  training  aids,  monitoring 
developments  in  training  technology,  preparing  examinations,  developing 
internal  feedback  programs,  evaluating  instructors,  conducting  annual  course 
reviews,  and  conducting  special  research  projects.  Five  or  more  CISOs  report 
no  involvement  with  task  analysis,  maintaining  a  central  technical  library, 
maintaining  a  test  item  bank,  and  coordinating  interservice  training 
requirements. 

10.  It  was  not  possible  to  determine  exactly  how  many  actual  CISO 
billets  there  were  in  the  NAVEDTRACOM.  Numbers  of  personnel  changed  from 

week  to  week,  according  to  CIS  officers.  Transient  personnel  might  be  assigned 
to  CISO  billets  for  varying  periods  of  time,  or  several  personnel  might 
share  one  CISO  billet  over  a  period  of  time. 

11.  Over  80  percent  of  the  CISOs  are  staff  units  within  local  training 
activities;  the  remainder  are  line  units.  The  typical  CIS  officer  reports 
to  the  executive  officer.  Some  CIS  officers  report  to  positions  one  or  two 
levels  below  the  executive  officer.  CISOs  usually  have  two  branches,  although 
several  have  only  one  and  a  few  offices  have  three  or  four.  Approximately 

35  percent  of  the  CIS  officers  are  also  either  directors  of  training  or 
training  department  heads.  Also,  about  20  percent  of  the  CIS  officers  are 
civilian.  Finally,  the  typical  rank  for  a  CIS  officer  is  0-4,  although 
they  range  from  0-2  to  0-6. 
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12.  CISO  personnel  were  29  percent  civilian  education  or  training 
specialists,  12  percent  military  instructors  or  curriculum  writers,  and  the 
remainder  were  in  miscellaneous  categories.  Military  officers  were  25  percent 
of  the  respondents;  38  percent  were  enlisted;  and  35  percent  were  civilians. 

13.  More  than  half  of  the  CISO  personnel  had  at  least  a  bachelor's 
degree.  About  30  percent  had  post-graduate  degrees,  mostly  in  education  or 
social  science.  For  those  people  with  training  in  education,  the  weakest 

(in  terms  of  amount  of  training)  areas  were  tests  and  measurement,  and  evalua¬ 
tion. 


14.  The  two  most  important  resource  areas  in  which  CISOs  could  be 
augmented  concern  numbers  of  personnel  and  the  skill  and  training  of  personnel. 

15.  The  most  useful  short  courses  or  seminars  for  CISO  personnel  would 
be  in  curriculum  writing,  course  evaluation,  instructional  design,  and  test 
development. 

16.  Six  variables  show  moderate  relationships  with  the  ratings  that 
CISOs  received  as  indicated  by  correlational  analyses.  The  proportion  of 
personnel  who  communicate  with  the  Deputy  CIS  officer  concerning  technical 
issues  is  positively  related  to  rating.  The  proportion  of  CISO  personnel 
who  receive  tasking  from  outside  the  CISO  and  the  proportion  of  personnel 
who  communicate  with  CNTECHTRA  concerning  administrative  issues  are  both 
negatively  related  to  CISO  ratings.  Also,  the  higher  the  degree  of  involve¬ 
ment  and  the  greater  the  frequency  of  involvement  with  CISDEV  (see  4,  page 
21)  functions,  the  higher  the  rating  received.  Greater  frequency  of  involve¬ 
ment  with  CISWRK  functions  (see  4,  page  21)  was  also  related  to  higher  ratings. 
These  findings  may  be  interpreted  as  follows: 

•  CISO  personnel  should  be  primarily  responsible  to  the  CIS  officer 
rather  than  to  other  local  command  personnel. 

•  The  CISO  should  be  primarily  involved  in  performing  duties  that 
require  expertise  in  educational  technology  rather  than  performing 
general  administrative  duties. 

•  CISO  personnel  should  be  frequently  involved  in  performing  training 
development  (i.e.,  course/module  design)  functions. 

•  The  CISO  should  have  primary  responsibility  for  performing  functions 
such  as  designing  and  selecting  instructional  delivery  systems, 
conducting  appropriate  inservice  training  for  assigned  staff  (e.g., 

in  ISD  procedures),  evaluating  training  department  inservice  programs, 
and  conducting  special  research  projects. 

17.  None  of  the  various  evaluation  function  scales  had  any  direct 
relationship  with  CISO  rating.  However,  EVAL2  (see  4,  page  21)  did  have  an 
indirect  relationship  to  rating  through  curriculum  and  instructional  support 
functions.  Thus,  the  evaluation  function  performed  by  the  CISOs  is  important. 
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The  remainder  of  this  section  presents  detailed  information  concerning 
the  treatment  of  questionnaire  data. 

QUESTIONNAIRE  FINDINGS 

The  major  findings  of  the  questionnaire  portion  of  the  CISO  survey  are 
presented  and  discussed  below. 

QUESTIONNAIRE  RETURN  STATISTICS.  A  total  of  377  questionnaires  (both  short 
and  long  forms)  were  distributed  to  30  CISOs.  Of  these,  23  were  returned 
as  surplus  (i.e.,  not  needed  for  number  of  personnel  assigned)  and  254  were 
usable,  giving  an  overall  response  rate  of  72  percent.  Table  1  provides 
summary  statistics  for  the  questionnaire  effort.  Table  2  shows  participa¬ 
tion  in  the  survey  by  activity. 

TABLE  1.  OVERALL  SUMMARY:  QUESTIONNAIRE  SURVEY 


QUESTIONNAIRE  OISTR  UTION 

Started  29  Sep  1980 

Last  mailout  29  Oct  1980 

Stopped  accepting  returns  19  Dec  1980 

SURVEY  STATISTICS 


Number  of  CISOs  surveyed:  30 

Number  of  quesitonnaires  distributed:  377 

Total  number  of  returns:  288 

Number  of  usable  returns:  254 

Number  returned  as  surplus:  23 

Return  rate  (usable):  72  percent 

RETURN  RATES  8Y  GROUP 


CO/XO  (28  out  of  30)  93  percent 
CIS  Officer.  (27  out  of  30)  90  percent 
Training  Oept  Heads  (79  out  of  98)  81  percent 
Other  CISO  personnel  (120  out  of  203)  59  percent 


MAJOR  FINDINGS:  SHORT  FORM  QUESTIONNAIRE.  The  major  findings  from  the 
short  form  questionnaire  pertain  to  the  importance  of  general  training  support 
functions,  who  should  perform  those  functions,  and  the  perceived  effectiveness 
of  CISOs. 

Training  Support  Functions.  Eleven  general  training  support  functions  were 
rank  ordered  by  commanding  or  executive  officers,  CIS  officers,  and  training 
department  heads.  Ranks  were  assigned  on  the  basis  of  the  perceived  importance 
of  the  function  to  assure  training  quality.  The  average  ranks  given  to 
each  of  the  functions  by  each  group  are  shown  in  table  3.  The  rankings  are 
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TABLE  2.  CISO  STUDY  PARTICIPATION  BY  ACTIVITY 


ACTIVITY  QUESTIONNAIRES 

DISTRIBUTED 

NUMBER  OF 
USABLE 

RETURNS 

PERCENT 

USABLE 

RETURNS 

NATTC  MEMPHIS* 

26 

17 

65 

SERVSCOLCOM  Orlando* 

20 

6 

30 

NAVGMSCOL  Dam  Neck* 

5 

5 

100 

FLECOMBATRACEN  Dam  Neck* 

18 

12 

67 

FLETRACEN  Norfolk* 

22 

15 

68 

FLEASWTRACENLANT  Norfolk* 

10 

9 

90 

NAVPHIBSCOL  Little  Creek* 

8 

5 

63 

SW0SC0LC0M  Newport* 

5 

3 

60 

NETC  Newport* 

11 

9 

82 

NAVJUSTSCOL  Newport* 

5 

0 

0 

NAVDIVESALVTRACEN  Panama  City* 

7 

4 

57 

NAVTECHTRACEN  Corry  Station* 

8 

5 

63 

NAVSUBSCOL  New  London* 

10 

8 

80 

NATTC  Lakehurst 

15 

9 

60 

NAVSCOLCECOFF  Pt  Hueneme 

4 

3 

75 

SERVSCOLCOM  DET  Chanute  AFB 

5 

5 

100 

FLEASWTRACENPAC  San  Diego* 

25 

21 

84 

NAVSUBTRACENPAC  Pearl  Harbor 

15 

13 

87 

NAVSCOLEOD  Indian  Head 

10 

9 

90 

SERVSCOLCOM  San  Diego* 

16 

16 

100 

SERVSCOLCOM  Great  Lakes* 

18 

10 

56 

NAVSCOLTRANSMAN  Oakland 

2 

2 

100 

NAVDAMCONTRACEN  Philadelphia* 

7 

5 

71 

NAVTECHTRACEN  Meridian 

9 

6 

67 

FLECOMBATRACEN  San  Diego 

8 

6 

75 

SUBTRAFAC  San  Diego 

10 

8 

80 

NAVTECHTRACEN  Treasure  Island* 

11 

10 

91 

COMBATSYSTECHSCOL  Mare  Island* 

8 

7 

88 

NAVPHIBSCOL  Coronado 

9 

9 

100 

NAMTRAGRU  Memphis* 

27 

17 

63 

Totals 

354 

254 

72** 

*  Indicates  personal  visits. 

**This  figure  is  the  overall  usable  return  rate  computed  as  the  number  of 
usable  returns  divided  by  the  :otal  number  of  questionnaires  distributed 
minus  the  number  returned  as  surplus.  The  average  return  rate  per  school 
was  approximately  74  percent. 
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based  on  the  means  of  the  priorities  assigned  to  each  function  by  the  three 
groups.  For  example,  the  mean  of  all  ranks  given  to  curriculum  development 
by  commanding  or  executive  officers  was  lower  than  the  mean  for  any  other 
function  ranked  by  this  group.  Therefore,  curriculum  development  was  given 
a  rank  of  1,  indicating  that  commanding  or  executive  officers,  on  the  average, 
considered  it  more  important  to  assure  training  quality  than  other 
functions. 

The  ranks  in  table  3  reflect  the  relative  order  of  importance  to 
training  quality  that  the  three  groups  of  respondents  assigned  to  these 
training  support  functions.  The  rank  order  correlations  (Kendall,  1962) 
among  the  three  sets  of  ratings  are  quite  high,  ranging  from  0.81  to  0.93, 
indicating  substantial  agreement  among  the  respondents  on  the  order  of 
importance  of  the  11  functions. 

Assignment  of  Functions.  Respondents  were  also  asked  to  indicate  their 
order  of  preference  for  which  of  five  activities  should  have  primary 
responsibility  for  the  11  training  support  functions.  The  orders  of 
preference  are  shown  in  table  4  for  the  six  most  highly  rated  functions. 

For  all  six  functions,  the  training  department  and  the  CISO  are  the  first 
and  second  choices,  respectively,  of  the  134  respondents.  These  first  two 
choices  are  actually  ranked  quite  closely,  while  CNTECHTRA  is  a  relatively 
distant  third  choice.  Thus,  commanding  or  executive  officers,  training 
department  heads  and  CIS  officers  clearly  prefer  local  responsibility  for 
the  accomplishment  of  these  six  functions. 


TABLE  4.  MEAN  PREFERRED  OROER  OF  PRIMARY  RESPONSIBILITY 
FOR  FUNCTIONS 


CISO 

Training 

Department 

CNET 

CNTECHTRA 

IPDC 

Curriculum  development 

2 

1 

5 

3 

4 

Curriculum  revision 

2 

1 

5 

3 

4 

Instructor  training 

2 

1 

4 

3 

5 

Internal  evaluation 

2 

1 

4 

3 

5 

Test  development  and 

2 

1 

5 

3 

4 

revision 

Instructor  evaluation 

2 

1 

4 

3 

5 

Average  difference  in 

mean  rank 

between  CISO 

and  TD: 

0.5 

Average  difference  in 

mean  rank 

between  CISO 

and  CNTECHTRA: 

2.1 
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Ratings  of  CISOs.  Ratings  of  CISOs  were  obtained  with  four  items  in  the 
small  questionnaire.  These  items  concerned  satisfaction  with  the  CISO  concept, 
positive  contributions  that  the  CISO  makes,  overall  effectiveness  of  the 
CISO,  and  the  impact  eliminating  the  CISO  would  have  on  the  quality  of  train¬ 
ing.  The  internal  consistency  reliability  (Nunnally,  1978)  of  the  four 
items  was  0.90,  which  is  quite  high  and  indicates  the  four  items  should  be 
taken  together  as  an  overall  index  of  rating.  Thus,  further  analysis  of 
CISO  ratings  was  performed  using  tne  mean  of  the  four  rating  items  as  a 
more  accurate  estimate  of  CISO  performance. 

The  next  step  in  the  analysis  of  CISO  ratings  consisted  of  determining 
what  differences  and  similarities  there  were  among  the  ratings  of  CISOs 
from  the  three  different  sources  (i.e.,  commanding  or  executive  officers, 

CIS  officers  and  training  department  heads).  Analysis  of  variance,  using 
rating  as  the  predicted  variable  and  type  of  respondent  as  the  predictor, 
did  show  significant  differences  amor-g  ratings  from  the  three  sources  (F  = 
16.46,  pc. 0001).  Based  on  Scheffe1'  (1959)  method  of  simultaneous  multiple 
comparisons,  training  department  heacs  rated  CISOs  significantly  lower  than 
did  either  commanding  or  executive  officers  and  CIS  officers  (p.c.01  for 
both  comparisons).  The  difference  between  ratings  from  commanding  or  execu¬ 
tive  officers,  and  CIS  officers  was  lot  significant  and  was  in  fact  quite 
small. 

Even  though  the  absolute  levels  of  ratings  were  different  for  training 
department  heads,  it  remained  to  be  seen  whether  or  not  the  rank  orders  of 
CISO  ratings  were  similar  for  all  three  groups  of  respondents.  Kendall's 
(1962)  coefficient  of  concordance  was  0.49  for  the  three  sources  of  judgment 
and  for  the  22  CISOs  on  which  there  was  complete  rating  information.  The 
coefficient  of  concordance,  which  can  range  in  value  from  0  to  1,  indicates 
agreement  among  a  set  of  judges.  A  value  of  0  would  indicate  no  agreement, 
while  a  value  of  1  would  indicate  perfect  agreement.  Thus,  there  is  moderate 
agreement  among  the  three  types  of  respondents  about  the  rank  orders  of 
CISOs.  Accordingly,  the  mean  rating  of  each  CISO  across  all  three  groups 
of  respondents  is  a  more  accurate  estimate  of  CISO  performance  than  the 
rating  received  from  any  one  group. 

The  final  step  in  checking  the  quality  of  the  CISO  rating  scale  involved 
determining  the  extent  to  which  the  scale  reflected  something  about  the 
CISO  itself  rather  than  simply  reflecting  the  random  views  of  individual 
respondents.  This  aspect  of  the  rating  scale  would  be  indicated  by  signifi¬ 
cant  differences  among  CISO  ratings  (Borgatta  and  Jackson,  1980).  An  analysis 
of  variance  using  ratings  from  all  three  types  of  respondents  as  the  dependent 
variable  and  CISO  as  the  independent  variable  was  performed.  There  were 
significant  differences  among  the  CISOs  (F  =  1.72,  p-c.03)  indicating  that 
it  is  appropriate  to  average  ratings  from  all  three  types  of  respondents  to 
get  an  overall  rating  that  reflects  a  true  CISO  characteristic. 

While  the  reliability  of  the  composite  scale  of  items  used  to  represent 
ratings  of  CISOs  can  be  estimated,  the  validity  of  the  rating  scale  cannot. 
There  is  no  external  criterion  for  (’.ISO  performance  that  can  be  used  to 
check  the  validity  of  the  rating  scale.  However,  evidence  indicates  that 
the  scale  is  internally  consistent,  that  the  three  different  sources  of 
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ratings  agree,  and  that  the  rating  scale  is  measuring  a  true  characteristic 
of  CISOs  and  not  merely  reflecting  random  individual  perceptions.  Therefore, 
it  is  reasonable  to  assume  that  tne  rating  scale  is  a  valid  indicator  of 
CISO  performance.  Given  that  assumption,  the  ratings  of  CISOs  will  be  used 
as  a  criterion  in  further  analysis  of  the  data.  Also,  certain  general  state¬ 
ments  can  be  made  about  the  current  performance  of  CISOs. 

Ratings  that  CISOs  received  ranged  from  2.4  to  4.9  with  a  mean  of  3.3, 
indicating  that  some  CISOs  are  performing  unsatisfactorily  while  others  are 
doing  an  outstanding  job.  Apparently,  there  is  room  for  improvement  in 
implementing  the  CISO  concept. 

MAJOR  FINOINGS:  LONG  FORM  QUESTIONNAIRE.  The  major  findings  from  the  long 
form  questionnaire  are  presented  and  discussed  below.  These  discussions 
concern  the  characteristics  of  current  CISO  personnel,  and  the  types  of 
functions  CISOs  perform.  Evidence  is  also  presented  for  the  existence  of 
different  types  of  CISOs. 

Characteristics  of  CISO  Personnel.  Of  the  CISO  personnel  who  responded  to 
the  questionnaire,  29  percent  were  civilian  education  specialists  or  training 
specialists;  12  percent  were  military  instructors  or  curriculum  writers 
(NEC  9502  or  9506).  The  remainder  represented  a  number  of  diverse  occupa¬ 
tional  specialties.  Military  officers  constituted  25  percent  of  the  respond¬ 
ents;  38  percent  were  enlisted  personnel;  35  percent  were  civilians,  and  2 
percent  did  not  identify  themselves. 

Only  12  percent  of  the  respondents  had  less  than  a  high  school  diploma. 

The  median  level  of  education  was  a  bachelor's  degree,  while  about  30  percent 
held  post  graduate  degrees.  Of  those  with  bachelor's  degrees,  37  percent 
were  in  education  and  27  percent  were  in  social  science.  Of  those  with 
graduate  degrees,  over  50  percent  were  in  education.  Of  the  people  who 
reported  either  university  or  military  training  in  various  aspects  of  educa¬ 
tion,  tests  and  measurement  and  evaluation  were  the  weakest  areas,  perhaps 
indicating  a  need  for  increasing  training  in  these  areas  or  for  hiring  per¬ 
sonnel  who  have  stronger  backgrounds  in  these  areas. 

Approximately  60  percent  of  the  respondents  reported  that  they  had 
attended  IT  school;  the  typical  amount  of  time  spent  in  instructor  training 
was  reported  as  4  weeks.  About  48  percent  of  the  respondents  attended  military 
management  seminars  and  about  40  percent  attended  training  seminars. 

Of  the  five  resource  areas  that  respondents  ordered  in  terms  of  priority, 
number  of  personnel  and  skill  and  training  of  personnel  were  ranked  higher 
than  available  materials,  equipment,  or  money. 

Types  of  Functions.  Several  statistical  analyses  were  performed  to  determine 
if  there  are  patterns  of  functions  which  CISOs  perform  and  the  nature  of 
their  involvement  with  these  functions.  The  statistical  procedures  used 
for  this  determination  are  described,  and  the  specific  numerical  results 
obtained,  are  presented  in  appendix  D.  The  general  results  are  presented 
below. 
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The  statistical  analyses  indicated  thit  five  groups  of  functions  are 
homogeneous  in  terms  of  degree  of  involvement  reported  by  CISO  personnel. 

That  is,  within  each  group  of  functions,  individuals  report  similar  degrees 
of  involvement.  The  five  groups  of  functions  derived  from  the  analyses  are 
shown  in  table  5.  These  five  groups  may  be  interpreted  to  represent  general 
task  groups  or  jobs  which  CISOs  perform. 

CISWRK  is  the  name  attached  to  the  first  group  of  functions  shown  in 
Part  I  ot  table  5,  since  these  functions  primarily  involved  routine  work  in 
curriculum  and  instructional  support  areas.  Nine  evaluation  functions  are 
labeled  EVAL1.  CISDEV  labels  a  group  of  five  functions  that  are  concerned 
primarily  with  long  term  developmental  activities.  EVAL2  comprises  two 
evaluation  functions.  The  fifth  "group,"  TECLIB,  contains  only  one  function, 
that  of  maintaining  the  central  technical  library. 

The  function  groups  were  used  to  form  five  function  scales.  The  proce¬ 
dures  used  and  the  psychometric  properties  of  these  function  scales  are 
presented  in  appendix  D.  The  five  scales  were  used  to  represent  degree  of 
involvement  with  general  groups  of  functions.  These  five  degree  of  involve¬ 
ment  scales  were  then  used  to  determine  if  there  are  different  types  of 
CISOs  in  terms  of  their  degree  of  involvement  with  the  five  groups  of  functions. 
Statistical  analyses  (see  appendix  D)  resulted  in  the  six  clusters  of  CISOs 
shown  in  table  6. 

Types  of  CISOs.  Three  of  the  involvement  with  functions  scales,  CISDEV, 

EVAL2,  and  TECLIB,  show  significant  differences  across  CISOs.  Thus,  these 
three  scales  were  used  to  determine  if  there  are  different  types  of  CISOs. 

It  is  already  apparent  that  CISOs  vary  considerably  with  respect  both  to 
what  they  do  and  how  well  they  do  it.  However,  there  may  not  be  29  or  30 
distinctly  different  CISOs.  Instead,  there  may  be  a  more  limited  number  of 
types  of  CISOs.  Statistical  analysis  (see  appendix  D)  showed  six  distinct 
types  of  CISOs.  These  types  or  clusters  of  CISOs  are  shown  in  table  7. 

Clusters  I  and  II  are  similar  in  terms  of  the  EVAL2  and  the  TECLIB 
functions  but  cluster  I  is  relatively  higher  on  CISDEV  than  cluster  II. 

Both  I  and  II  are  equivalent  to  clusters  III  and  IV  in  terms  of  EVAL2,  but 
I  and  II  show  less  involvement  with  the  TECLIB  function  than  do  III  and  IV. 
Additionally,  III  and  IV  are  similar  in  terms  of  the  EVAL2  and  CISDEV  func¬ 
tions,  but  VI  is  higher  than  V  on  the  TECLIB  function.  With  this  typology 
of  CISOs  established,  it  remains  to  determine  what  other  characteristics 
are  shared  by  CISOs  within  each  of  the  six  clusters. 

The  six  types  of  CISOs  do  not  differ  significantly  in  terms  of  involve¬ 
ment  with  the  functions  represented  by  the  scales  CISWRK  and  EVAL1.  All 
six  types  score  rather  close  to  the  scale  mid-point.  This  probably  indicates 
that  CISOs  are  spread  rather  evenly  in  terms  of  involvement  with  these  two 
groups  of  functions.  The  remaining  three  involvement  scales,  CISDEV,  EVAL2, 
and  TECLIB,  do  show  significant  differences  among  the  six  types  of  CISOs. 
However,  this  is  to  be  expected  since  these  scales  were  used  to  establish 
the  typology. 
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TABLE  5.  COMPONENTS  OF  FUNCTION  SCALES 


PART  I.  INVOLVEMENT  SCALES 


Check  the  response  that  best  describes  your  involvement  with 
each  function  listed. 

(1)  Not  involved 

(2)  Monitor  those  who  do  it 

(3)  Advise  those  who  do  it 

(4)  Participate  in  performance 

(5)  Supervise  those  who 
do  it 

(6)  Do  it  alone 


CISWRK  - 


EVAL1 


CISDEV 


EVAL2 


TECLIB 


Review  course  and  curricula  data  and  documentation 
Do  task  analysis 
Develop  curricula 
Design/revise  courses/curricula 
Develop  training  materials/aids 

Prepare  examinations 
Analyze  test  data 
Maintain  test  item  bank 

Develop  internal  feedback  instruments/procedures 
Develop  items  for  external  feedback  instruments 
Analyze  and  interpret  feedback  data 
Study  attrition 
Study  setbacks 

Administer  student  critique  program 

Monitor  developments  in  training  technology  to 
recommend  improvements  in  training 
Select  instructional  delivery  systems 
Develop/conduct  inservice  training 
Evaluate  training  department  inservice  programs 
Conduct  special  research  projects 

Evaluate  instructors,  including  contract  instructors 
Conduct  annual  course  review 

Maintain  central  technical  library 
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TABLE  5.  COMPONENTS  OF  FUNCTION  SCALES  (continued) 


II.  FREQUENCY  OF  PERFORMANCE  SCALES 


On  the  average,  how  often  do  you  perform  each  function? 


(1) 

Oai  ly 

(2) 

Weekly 

(3) 

Monthly 

(4) 

Every  3  months 

(5) 

Every  6  months 

(6) 

Yearly  or  less 

CSWRKB  - 

Review  course  and  curricula  data  and  documentation 

Do  task  analysis 

Develop  curricula 

Design/revise  courses/curricula 

Develop  training  materials/aids 

EVAL1B  - 

Prepare  examinations 

Analyze  test  data 

Maintain  test  item  bank 

Develop  internal  -feedback  instruments/procedures 
Develop  items  for  external  feedback  instruments 
Analyze  and  interpret  feedback  data 

Study  attrition 

Study  setbacks 

Administer  student  critique  program 

CSDEVB  - 

Monitor  developmerts  in  training  technology  to 
recommend  improvements  in  training 

Select  instructional  delivery  systems 

Oevelop/conduct  irservice  training 

Evaluate  training  department  inservice  programs 
Conduct  special  research  projects 

EVAL2B 

Evaluate  instructers,  including  contract  instructors 
Conduct  annual  coirse  review 

TCLIBB 


Maintain  central  technical  library 
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TABLE  5.  COMPONENTS  OF  FUNCTION  SCALES  (continued) 


PART  III.  AMOUNT  OF  EFFORT  SCALES 


For  the  functions  you  are  involved  with,  check  the  response  that  best 
describes  the  amount  of  effort  you  put  in. 


(1)  A  very  slight  amount 

(2)  A  slight  amount 

(3)  A  moderate  amount 

(4)  A  considerable  amount 

(5)  A  great  amount 


CSWRKC 


EVAL1C 


CSDEVC 


EVAL2C 

TCLIBC 


Review  course  and  curricula  data  and  documentation 
Do  task  analysis 
Develop  curricula 
Design/revise  courses/curricula 
Develop  training  materials/aids 

Prepare  examinations 
Analyze  test  data 
Maintain  test  item  bank 

Develop  internal  feedback  instruments/procedures 
Develop  items  for  external  feedback  instruments 
Analyze  and  interpret  feedback  data 
Study  attrition 
Study  setbacks 

Administer  student  critique  program 

Monitor  developments  in  training  technology  to 
recommend  improvements  in  training 
Select  instructional  delivery  systems 
Develop/conduct  inservice  training 
Evaluate  training  department  inservice  programs 
Conduct  special  research  projects 

Evaluate  instructors,  including  contract  instructors 
Conduct  annual  course  review 

Maintain  central  technical  library 
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TABLE  5.  COMPONENTS  OF  FUNCTION  SCALES  (continued) 


PART  IV.  IMPACT  OF  ELIMINATION  SCALES 

For  each  function  with  which  you  are  involved,  how  much  impact 
would  there  be  on  the  quality  of  courses  if  this  function  were  no 
longer  performed  by  CISO  personnel? 

(1)  No  impact 

(2)  Little  impact 

(3)  Some  impact 

(4)  Considerable  impact 

(5)  A  great  deal  of  impact 


CSWKVD 


EVL1VD 


CSDVVD- 


EVL2VD- 


Review  course  and  curricult  data  and  documentation 

Do  task  analysis 

Develop  curricula 

Design/revise  courses/curr- cula 

Develop  training  rraterials/aids 

Prepare  examinations 
Analyze  test  data 
Maintain  test  item  bank 

Develop  internal  feedback  -instruments/procedures 
Develop  items  for  external  feedback  instruments 
Analyze  and  interpret  feedback  data 
Study  attrition 
Study  setbacks 

Administer  student  critique  program 

Monitor  developments  in  training  technology  to 
recommend  improvements  in  training 
Select  instructional  delivery  systems 
Develop/conduct  inservice  training 
Evaluate  training  department  inservice  programs 
Conduct  special  research  projects 

Evaluate  instructors,  including  contract  instructors 
Conduct  annual  course  review 


TCLIBD 


Maintain  central  technical  library 
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TABLE  5.  COMPONENTS  OF  FUNCTION  SCALES  (continued) 


PART  V.  POSITIVE/NEGATIVE  IMPACT  SCALE 

Would  the  impact  of  the  elimination  of  this  function  be 

positive 

or  negative? 

(1) 

Positive 

(2) 

Negative 

CSWKVE- 

Review  course  and  curricula  data  and  documentation 

Do  task  analysis 

Oevelop  curricula 

Oesign/revise  ccurses/curricula 

Develop  training  materials/aids 

EVL1VE 

Prepare  examinations 

Analyze  test  data 

Maintain  test  item  bank 

Develop  internal  feedback  instruments/procedures 

Develop  items  for  external  feedback  instruments 

Analyze  and  interpret  feedback  data 

Study  attrition 

Study  setbacks 

Administer  student  critique  program 

CSDVVE 

Monitor  developments  in  training  technology  to 
recommend  improvements  in  training 

Select  instructional  delivery  systems 

Develop/conduct  inservice  training 

Evaluate  training  department  inservice  programs 

Conduct  special  research  projects 

EVL2VE 

Evaluate  instructors,  including  contract  instructors 
Conduct  annual  course  review 

TCLIBE 

Maintain  central  technical  library 
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CLUSTER 


CLUSTER 


CLUSTER 


CLUSTER 


CLUSTER 


CLUSTER 


TABLE  6.  COMPOSITION  OF  CLUSTERS 


I  -  NAVGMSCOL 

NAVTECHTRACEN 
NAVTECHT'RACEN 
FLECOMBATRACEN 
COMBATS VSTECHSCOL 
NAVPHIBSCOL 


II  -  NATTC 

FLEASWTRACENPAC 

NAVSUBTUACENPAC 

NAVSCOLLOO 

SERVSCOI.COM 

SUBTRAFAC 


III  -  NAVPHIBSCOL 
NETC 

SERVCOLCOM 

NAVDAMCONTRACEN 


IV  -  NAVDIVE  SALVTRACEN 

NAVSCOLOECOFF 
SERVSCOI.COM  OET 
NAVSCOL  "RANSMAN 


V  -  NATTC 

FLETRACEN 

SWOSCOL 

NAVSUBSCOL 

NAMTRAGRU 


VI  -  SERVSCOLCOM 

FLECOMBATRACEN 

FLEASWTRACENLANT 

NAVTECHTRACEN 


DAM  NECK 
CORRY  STATION 
MERIDIAN 
SAN  DIEGO 
MARE  ISLANO 
CORONADO 


LAKEHURST 
SAN  DIEGO 
PEARL  HARBOR 
INDIAN  HEAD 
SAN  DIEGO 
SAN  DIEGO 


LITTLE  CREEK 
NEWPORT 
GREAT  LAKES 
PHILADELPHIA 


PANAMA  CITY 
PT.  HUENEME 
CHANUTE  AFB 
OAKLAND 


MEMPHIS 
NORFOLK 
NEWPORT 
NEW  LONDON 
MEMPHIS 


ORLANDO 
DAM  NECK 
NORFOLK 

TREASURE  ISLANO 
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Since  the  list  of  functions  is  identical  for  all  questions  asked  about 
performance,  the  same  groupings  of  functions  were  retained  for  questions  B 
and  C  in  section  IV  and  questions  D  and  E  of  section  V  of  the  long  question¬ 
naire  (appendix  B).  This  allows  for  direct  comparisons  among  the  five  groups 
of  functions  and  the  six  types  of  CISOs  for  the  frequency  of  performance 
(Part  II,  table  5),  the  impact  of  elimination  (Part  IV,  table  5),  and  the 
positive  or  negative  aspect  of  elimination  (Part  V,  table  5).  The  discriminat¬ 
ing  variables,  CSWRKB  through  TCLIBE  (shown  in  table  7),  are  explained  in 
Parts  II  through  V  of  table  5. 

Table  7  shows  the  relative  ranks  of  the  six  clusters  of  CISOs  on  the 
function  scales  and  other  variables  that  differentiate  among  the  CISOs.  A 
'+'  indicates  that  the  cluster  is  relatively  high  on  the  variable;  a  'O' 
indicates  that  the  cluster  is  medium,  and  a  '-'  indicates  that  a  cluster  is 
low.  For  example,  clusters  I  and  IV  are  relatively  high  on  the  CISDEV  func¬ 
tion  scale,  while  cluster  VI  is  low  and  clusters  II,  III,  and  V  are  medium. 

Other  variables  from  the  questionnaire  are  also  displayed  in  table  7. 
Clusters  I  and  III  are  high  in  the  percent  of  personnel  with  undergraduate 
degrees  in  social  science  or  technical  or  scientific  subjects.  Cluster  II 
is  high  in  the  percent  of  senior  enlisted  personnel  (E6-E9)  assigned  to 
CISO,  while  cluster  I  is  high  in  civilians  (GS9-GS13),  and  cluster  III  is 
high  in  percent  of  officers.  The  relative  positions  of  CISO  clusters  on 
the  other  variables  can  be  determined  by  examining  table  7. 

The  clusters  or  types  of  CISOs  can  be  given  labels  that  reflect  their 
nature  relative  to  each  other.  For  example,  the  CISOs  in  cluster  I  are 
relatively  heavily  involved  in  developmental  curriculum  and  instructional 
support  functions  (CISDEV),  and  frequently  perform  working  curriculum  and 
instructional  support  functions  (CSWRKB)  and  evaluation  functions  (EVAL2B). 

The  personnel  in  cluster  I  CISOs  tend  to  be  relatively  highly  educated  civil¬ 
ians.  CISOs  in  cluster  II  are  also  frequently  involved  in  working  CIS  func¬ 
tions,  but  their  personnel  have  less  education  and  are  more  likely  to  be 
enlisted  military. 

Cluster  III  CISOs  are  most  notable  for  their  degree  of  involvement 
with  maintaining  a  central  technical  library;  they  report  relatively  infre¬ 
quent  involvement  with  both  curriculum  and  instructional  support  and  evalua¬ 
tion  functions.  These  CISOs  have  a  higher  percentage  of  officers  and  assigned 
personnel  have  higher  levels  of  education. 

Cluster  IV  reports  the  deepest  degree  of  involvement  with  the  most 
functions.  Also,  these  CISOs  report  frequent  involvement  with  the  develop¬ 
mental  CIS  functions.  These  CISOs  have  personnel  who  are  relatively  highly 
educated  and  who  have  attended  military  management  seminars  to  a  greater 
extent  then  have  other  CIS  personnel. 

Cluster  V  CISOs  are  distinguished  primarily  by  the  large  numbers  of 
courses  serviced  and  by  number  of  personnel,  as  indicated  by  number  of 
respondents.  These  personnel  also  report  greater  attendance  at  military 
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management  seminars  and  relatively  frequent  performance  of  working  CIS  func¬ 
tions  (CSWRKB).  CISOs  in  cluster  VI  report  higher  involvement  with  the 
library  but  low  involvement  with  all  other  functions. 

In  summary,  there  are  different  types  of  CISOs.  Evidence  suggests 
that  there  are  six  types  that  differ  in  terms  of  functions  performed  and  in 
terms  of  personnel  characteristics. 

Ratings  of  Clusters.  The  mean  ratings,  from  commanding  or  executive  officers, 
CIS  officers  and  training  department  heads,  of  the  CISO  clusters  (shown  in 
the  row  labeled  "Rating"  in  table  7)  differ  significantly  based  on  an  ANOVA 
(F  =  2.6,  p<_.05).  Clusters  IV  and  V  have  the  highest  rating,  clusters  I 
and  II  have  medium  ratings,  and  clusters  III  and  VI  have  relatively  low 
ratings.  The  order  of  the  clusters,  in  terms  of  mean  rating,  is  indicated 
by  the  figures  in  parentheses  in  the  ratings  columns.  However,  some  of  the 
differences  in  means  are  quite  small.  For  example,  clusters  I  and  II  differ 
by  less  than  0.001,  so  it  is  not  appropriate  to  say  that  cluster  I  is  better 
than  cluster  II.  But  cluster  IV,  the  highest  rated,  does  appear  to  be  con¬ 
siderably  higher  than  cluster  III,  the  lowest  rated.  Therefore,  it  may  be 
instructive  to  compare  the  profiles  of  extreme  clusters  to  look  for  character¬ 
istics  that  may  be  related  to  the  differences  in  ratings  received. 

A  quick  glance  shows  that  cluster  IV  has  more  +'s  and  fewer  -'s  than 
cluster  III.  However,  that  is  also  true  when  IV  is  compared  to  V,  the  second 
highest  rated  cluster.  There  are  some  characteristics  that  IV  and  V  share 
but  that  III  does  not.  For  example,  cluster  III  is  low  on  the  frequency  of 
performance  of  CSWRKB  and  CSDEVB  functions  while  clusters  IV  and  V  are  high 
or  medium.  Also,  clusters  IV  and  V  are  high  on  the  number  of  hours  personnel 
have  spent  attending  military  management  seminars,  while  cluster  III  is 
low.  Finally,  cluster  IV  contains  the  smallest  CISOs,  both  in  terms  of 
number  of  respondents  and  reported  number  of  courses  serviced,  while  cluster 
V  contains  the  largest  CISOs.  Thus,  one  might  say  that  it  is  best  to  be  a 
very  large  CISO,  perhaps  because  they  have  a  valuable  flexibility  in  allocating 
personnel  and  resources,  or  to  be  a  very  small  CISO,  perhaps  because  they 
can  establish  close  working  relationships  with  schoolhouse  personnel. 

Several  general  statements  can  be  made  about  CISOs  at  this  point.  First, 
there  do  appear  to  be  six  distinct  types  of  CISOs  that  vary  on  a  variety  of 
variables,  including  rated  effectiveness.  Second,  the  rated  effectiveness 
of  these  clusters  of  CISOs  appears  to  be  related  to  several  discriminating 
characteristics.  It  may  be  that  the  deliberate  manipulation  of  some  of 
these  characteristics  could  result  in  improving  the  effectiveness  of  CISOs 
in  the  less  highly  rated  clusters. 

Involvement  With  Functions.  The  29  CISOs  surveyed  differ  in  their  level  of 
involvement  with  functions.  They  also  differ  in  terms  of  the  number  of 
functions  that  they  do/do  not  perform.  Table  8  presents  a  list  of  training 
activities  and  the  numbers  of  functions  that  CIS  personnel  report  no  involve¬ 
ment  with  at  all.  If  an  activity  does  not  appear  in  the  table,  then  at 
least  one  person  in  the  CISO  reported  at  least  minimal  involvement  (i.e., 
monitor  those  who  do  it)  with  all  of  the  24  functions.  Thus,  this  is  an 
extremely  conservative  estimate  of  lack  of  CISO  involvement  with  functions 
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TABLE  8.  NUMBER  OF  FUNCTIONS  NOT  PERFORMED  BY  CISOs 


NUMBER  OF  FUNCTIONS 

3 

4 
12 

6 

1 

1 

1 

2 

2 

4 

5 
2 

6 
1 
2 


CISOs 

SERVSCOLCOM  Orlando 
FLEASWTRACEN  Norfolk 
NAVPHIBSCOL  Little  Creek 
SWOSCOL  Newport 
NAVDIVESALVTRACEN  Panama  City 
NAVTECHTRACEN  Corry  Station 
NAVSCOLCECOFF  Pt  Hueneme 
NAVSUBTRACENPAC  Pearl  Harbor 
NAVSCOLEOD  Indian  Head 
NAVSCOLTRANSMAN  Oakland 
NAVDAMCONTRACEN  Philadelphia 
FLECOMBATRACEN  San  Diego 
SUBTRAFAC  San  Diego 
NAVTECHTRACEN  Treasure  Island 
COMBATSYSTECHSCOL  Mare  Island 


1 


NAVPHIBSCOL  Coronado 
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specified  in  instructions.  The  number  of  functions  with  which  a  CISO  is 
not  involved  ranges  from  12  to  0.  Additionally,  no  significant 
relationship  exists  between  the  number  of  functions  that  a  CISO  is  not 
involved  with  and  the  rating  that  a  CISO  received  from  commanding  or 
executive  officers,  CIS  officers,  and  training  department  heads. 

Numbers  of  functions  not  performed  by  particular  CISOs  indicate  the 
variability  in  the  current  implementation  of  the  CISO  concept.  The  numbers 
of  CISOs  that  are  not  involved  with  particular  functions  may  indicate  the 
relative  importance  of  the  various  functions.  Table  9  displays  the  24  func¬ 
tions  under  consideration  and  the  number  of  CISOs  (out  of  the  28  for  which 
data  are  complete)  reporting  no  involvement  with  those  functions.  Again, 
the  criterion  for  involvement  was  that  at  least  one  person  in  the  CISO  reported 
at  least  the  minimum  degree  of  involvement.  Out  of  the  24  functions,  there 
are  only  five  with  which  all  CISOs  report  some  degree  of  involvement.  Further, 
there  are  six  functions  with  which  at  least  four  CISOs  report  no  involvement 
whatsoever. 

In  summary,  when  one  examines  the  picture  of  functions  that  CISOs  do 
and  do  not  perform,  the  most  striking  conclusion  is  that  CISOs  are  implemented 
in  various  ways.  There  may  be  no  single  best  way  to  implement  the  CISO 
concept  across  all  training  activities.  While  there  may  be  a  core  of  functions 
that  is  desirable  for  all  CISOs  to  perform,  some  flexibility  in  the  means 
used  to  achieve  those  functions  may  be  desirable. 

The  variation  in  the  considered  importance  of  certain  functions  is 
also  reflected  in  the  numbers  of  CISOs  that  report  little  or  no  expected 
impact  from  eliminating  a  particular  function  (table  10).  The  criterion 
used  was  that  every  respondent  in  a  CISO  reported  little  or  no  expected 
impact  from  eliminating  each  function,  a  very  conservative  decision  rule. 

Using  this  criterion,  only  six  functions  on  the  list  were  considered  necessary 
by  at  least  one  respondent  in  all  the  CISOs.  Again,  the  point  can  be  made 
that  all  24  of  these  functions  are  probably  not  necessary  for  every  CISO  to 
perform,  although  there  is  a  core  of  functions  that  is  universally  necessary. 

PREDICTORS  OF  RATINGS.  The  final  phase  of  analysis  examined  the 
relationships  between  ratings  of  CISOs  and  various  predictors  for  the  28 
CISOs  for  which  data  are  complete.  The  predictors  were  all  those  available 
that  showed  statistical  evidence  that  they  represented  true  CISO-level 
characteristics.  Only  variables  that  did  differ  significantly  among 
schools  were  used,  since  it  is  only  these  variables  that  could  be  related 
to  CISO  ratings  in  a  meaningful  way  (Borgatta  and  Jackson,  1980). 

Six  of  the  variables  showed  moderate  relationships  with  CISO  rating, 
with  correlations  ranging  from  0.3  to  0.5  in  magnitude.  Three  of  the 
variables  that  predict  CISO  ratings  are  concerned  with  degree  of 
involvement  and  frequency  of  Involvement  with  functions.  These  three 
variables,  CISDEV,  CSWRKB,  and  CSDEVB,  are  explained  in  table  5,  page  31. 

As  shown  in  table  11,  their  relationships  with  rating  are  all  positive. 

The  remaining  three  variables  that  predict  rating  are  concerned  with 
patterns  and  content  of  communication  for  CISO  personnel.  These  variables 
are  all  based  on  items  from  section  III  of  the  large  questionnaire  (see 
appendix  B). 
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TABLE  9.  NUMBER  OF  CISOs  NOT  INVOLVED  WITH  FUNCTIONS 


FUNCTIONS  NUMBER  OF  CISOS 

NOT  INVOLVED 

Review  course  and  curricula  data  and 
documentation 

0 

Do  task  analysis 

5 

Develop  curricula 

0 

Design/devise  courses/curricula 

0 

Develop  training  aids/materials 

0 

Monitor  developments  in  training  technology 

1 

Select  instructional  delivery  syftems 

3 

Develop/conduct  inservice  training 

2 

Maintain  central  technical  library 

5 

Prepare  examinations 

1 

Analyze  test  data 

2 

Maintain  test  item  bank 

7 

Develop  internal  feedack  instruments/procedures 

1 

Develop  items  for  external  f eedb vek 
instruments 

2 

Analyze  and  interpret  feedback  d.ita 

2 

Study  attrition 

4 

Study  setbacks 

3 

Administer  student  critique  program 

2 

Evaluate  instructors,  including  contract 
instructors 

1 

Evaluate  training  department  inservice 
programs 

2 

Conduct  annual  course  reviews 

0 

Conduct  special  research  project'. 

1 

Coordinate  interservice  training  requirements 

5 

Coordinate  interdepartmental  training  activities 

4 
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TABLE  10.  NUMBER  OF  CISOs  REPORTING  LITTLE  OR  NO  IMPACT 
OF  ELIMINATION  OF  FUNCTIONS 


FUNCTIONS  NUMBER  OF  CISOS 

Review  course  and  curricula  data  and  0 

documentation 

Do  task  analysis  5 

Develop  curricula  0 

Design/devise  courses/curricula  0 

Develop  training  aids/materials  3 

Monitor  developments  in  training  technology  2 

Select  instructional  delivery  systems  2 

Develop/conduct  in-service  training  1 

Maintain  central  technical  library  6 

Prepare  examinations  1 

Analyze  test  data  0 

Maintain  test  item  bank  3 

Develop  internal  evaluation  instruments/procedures  1 

Develop  items  for  external  feedback  3 

instruments 

Analyze  and  interpret  feedback  data  0 

Study  attrition  4 

Study  setbacks  4 

Administer  student  critique  program  4 

Evaluate  instructors,  including  contract  1 

instructors 

Evaluate  training  department  in-service  3 

programs 

Conduct  annual  course  reviews  0 

Conduct  special  research  projects  2 

Coordinate  interservice  training  requirements  3 

Coordinate  interdepartmental  training  activities  1 
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SCALE 

DEPTEC 

(+) 

TASK IN 
(-) 

CNADMN 

(-) 

CISDEV 

(+) 

CSWRKB 

(+) 

CSDEVB 

(+) 


TABLE  11.  BEST  PREDICTORS  OF  CISO  RATING 


Proportion  of  CIS  personnel  communicating 
with  Deputy  CIS  officer  concerning  technical 
issues 


Proportion  of  CIS  personnel  who  receive 
tasking  from  outside  the  CISO 


Proportion  of  CIS  personnel  who  communicate 
with  CNTECHTRA  concerning  administrative 
issues 


Degree  of  involvement  with  developmental 
curriculum  and  instructional  support 
functions 


Frequency  of  involvement  with  working 
curriculum  and  instructional  support 
functions 


Frequency  of  involvement  with  developmental 
curriculum  ard  instructional  support 
functions 
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DEPTEC  represents  the  proportion  of  CISO  personnel  who  communicate 
with  the  Deputy  CIS  officer  concerning  technical  issues;  it  is  positively 
related  to  CISO  ratings.  TASKIN  represents  the  proportion  of  CISO  personnel 
who  receive  tasking  from  outside  the  CISO;  it  is  negatively  related  to 
CISO  rating.  That  is,  as  more  CISO  personnel  receive  tasking  from  outside 
the  CISO,  the  lower  that  office's  rating  is  likely  to  be.  CNADMN  represents 
the  proportion  of  CISO  personnel  who  communicate  with  CNTECHTRA  concerning 
administrative  issues;  it  is  also  negatively  related  to  CISO  rating. 

There  appears  to  be  a  coherent  picture  in  the  six  variables  that  predict 
CISO  ratings.  First,  those  CISOs  that  use  the  deputy  CIS  officer,  or  the 
senior  education  specialist,  as  a  source  of  professional  technical  information 
and  advice,  tend  to  receive  higher  ratings.  This  is  indicated  by  the  positive 
relationship  between  DEPTEC  and  the  CISO  ratings.  Further,  this  relationship 
supports  the  view  that  the  principal  contribution  the  CISO  has  to  make  to 
local  training  activities  is  as  the  repository  of  skills  and  knowledge  concern¬ 
ing  educational  technology.  It  can  also  be  argued  that  this  should  be  the 
principal  contribution  that  the  CISO  makes  to  the  local  training  command. 

This  is  evidenced  by  the  negative  relationships  between  TASKIN  and  ratings 
and  between  CNADMN  and  ratings.  Thus,  CISO  personnel  should  be  responsible 
primarily  to  the  CIS  officer,  not  to  other  personnel.  They  should  involve 
themselves  with  CISO  duties  that  reflect  their  expertise  in  educational 
technology  and  should  not  become  bogged  down  in  general  administrative  duties. 

In  addition  to  addressing  optional  lines  of  authority  and  communication, 
it  is  also  possible  to  describe  the  general  types  of  functions  with  which 
CISOs  should  be  involved.  Both  degree  of  involvement  with  and  frequency  of 
involvement  with  the  developmental  curriculum  and  instructional  support 
functions  (see  CISDEV  in  table  5  for  a  list  of  those  functions)  are  positively 
related  to  CISO  ratings.  Therefore,  some  effort  in  CISOs  should  probably 
be  directed  toward  these  developmental  functions. 

Frequency  of  involvement  with  working  curriculum  and  instructional 
support  functions  (see  CISWRK  in  table  5  for  a  list  of  those  functions)  is 
also  positively  related  to  rating.  Therefore,  frequent  involvement  in 
performing  these  functions  is  desirable.  However,  the  degree  of 
involvement  should  be  moderate  and  probably  should  consist  primarily  of 
advice  and  assistance. 

None  of  the  various  evaluation  scales  showed  any  significant  relation¬ 
ships  with  CISO  ratings.  However,  EVAL2  (see  table  5)  did  show  a  strong 
relationship  with  CISDEV,  which  is  positively  related  to  rating.  Thus, 
there  appears  to  be  a  multivariate  relationship  among  CISDEV,  EVAL2,  and 
rating.  Path  analysis  (Kenny,  1979)  was  used  to  explore  possible  relation¬ 
ships  among  these  three  variables.  The  results  of  the  path  analysis  are 
shown  in  figure  1.  The  arrows  indicate  the  expected  directions  of  effect. 

Thus,  both  CISDEV  and  EVAL2  lead  to  or  have  an  impact  on  the  ratings  that 
CISOs  receive,  but  CISDEV  and  EVAL2  only  vary  together.  That  is,  as  one 
goes  up  or  down,  so  does  the  other,  but  there  is  no  necessary  cause  and 
effect  relationship  between  them.  The  numbers  by  the  arrows  in  figure  1 
are  standardized  path  coefficients  and  may  be  interpreted  similarly  to  cor¬ 
relation  coefficients.  Values  may  range  from  -1  to  +1  and  the  magnitude  of 
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the  coefficient  indicates  the  strength  of  the  relationship  while  the  sign 
indicates  the  direction. 

Based  on  the  path  diagram  in  figure  1,  CISDEV  has  a  moderate  positive 
impact  on  rating,  while  EVAL2  has  a  very  small  negative  impact  on  rating. 
However,  even  though  the  direct  impact  of  EVAL2  on  ratinq  is  negligible, 
the  indirect  impact  through  CISDEV  is  appreciable.  There  is  some  logic  to 
this  interpretation.  There  is  no  reason  to  expect  the  performance  of  evalua¬ 
tion  functions  to  have  any  direct  impact  on  ratings.  Evaluation,  in  and  of 
itself,  does  not  cause  any  improvement.  It  is  only  when  information  obtained 
from  evaluation  is  acted  on  that  any  contributions  are  made  to  accomplishing 
local  training  goals.  Thus,  even  though  evaluation  functions  do  not  impact 
directly  on  CISO  ratings,  there  does  appear  to  be  an  indirect,  but  nonethe¬ 
less  important,  relationship  between  evaluation  and  ratings. 

ORGANIZATIONAL  CHARACTERISTICS  OF  CISOs 

Organizational  characteristics  of  local  CISOs  examined  during  the  study 
included  CISO  size  (i.e.,  number  of  personnel)  and  the  formal  relationship 
of  the  CISO  to  the  local  training  activity.  These  results  are  based  on 
both  questionnaire  data  and  on  the  analysis  of  organization  charts  obtained 
from  CISOs. 

Size.  Size  is  typically  one  of  the  most  straightforward  organizational 
characteristics  to  measure.  However,  in  the  case  of  CISO,  size  was  difficult 
to  determine.  Billets  are  often  not  designated  as  CISO  billets  but  merely 
as  curriculum  writers.  Additionally,  one  CISO  billet  may  be  rotated  among 
several  personnel  over  a  period  of  time.  In  other  cases,  transient  personnel 
may  be  assigned  to  CISO  billets  for  various  periods  of  time.  Also,  non- 
CISO  personnel  may  perform  CISO  functions.  Thus,  a  meaningful  assessment 
of  size  was  not  possible. 

Formal  Organization  of  CISOs.  To  assess  other  organizational  characteris¬ 
tics  of  CISOs,  the  organization  charts  obtained  from  each  training  activity 
and  information  obtained  during  visits  to  CISOs  were  used.  From  these,  it 
was  possible  to  determine  six  organizational  characteristics  of  CISOs: 

•  whether  the  office  served  a  line  or  staff  function 

•  to  whom  the  CIS  officer  reported  in  the  training  activity 

•  the  number  of  branches  in  the  CISO 

•  whether  the  CIS  officer  was  also  a  training  director  or  a  training 
department  head 

•  whether  the  CIS  officer  was  civilian  or  military 

•  the  rank  of  th»  CIS  officer. 
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Figure  1 


Path  Analysis  of  the  Effects  of  Curriculum  and  Instructional  Support 
and  Evaluation  Functions  on  Ratings 
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Over  80  percent  of  the  CISOs  are  staff  units  within  local  training 
activities;  the  remainder  are  line  units.  The  typical  CIS  officer  reports 
to  the  executive  officer,  while  some  report  to  positions  one  or  two  levels 
below  the  executive  officer.  CISOs  usually  have  two  branches,  although 
several  have  only  one  and  a  few  offices  have  three  or  four.  Approximately 
35  percent  of  the  CIS  officers  are  also  either  directors  of  training  or 
training  department  heads.  About  20  percent  of  the  CIS  officers  are  civilian 
and  the  typical  rank  for  a  military  CIS  officer  is  0-4,  although  they  ranqe 
from  0-2  to  0-6. 
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SECTION  V 

RESULTS  AND  DISCUSSION  OF  CISO  EVALUATION: 

INTERVIEWS  WITH  LOCAL  TRAINING  ACTIVITY  PERSONNEL 

This  section  presents  and  discusses  information  obtained  from 
interviews  with  CISO  personnel  and  other  local  training  activity  personnel. 

FACTORS  AFFECTING  CISO  PERFORMANCE 

Information  presented  in  the  previous  section  demonstrated  that  local 
CISOs  are  quite  variable  organizationally  and  in  terms  of  tasks  performed. 
Interviews  with  local  CISO  staffs  and  other  command  personnel  centered  on 
factors  which  influence  local  assignment  of  training  quality  functions  to 
CISOs  and/or  otherwise  operate  1o  limit  the  success  (in  terms  of  enhancing 
the  quality  of  training)  achievable  by  CISOs.  A  number  of  such  factors 
were  discussed.  They  are  complex,  multifaceted  and  strongly  interrelated. 

They  do  not  operate  in  standard  or  uniform  ways  across  all  CISOs.  Neverthe¬ 
less,  they  represent  both  current  and  future  problem  areas  which  require 
attention.  Modification  or  alternatives  to  the  current  CISO  situation  should 
consider  ways  of  minimizing  the  impact  of  these  problems  in  the  interest  of 
promoting  training  quality.  The  remainder  of  this  section  discusses  factors 
which  affect  local  CISO  operations  and  potential  effectiveness. 

GENERAL  DISCUSSION.  By  and  large,  local  commands  have  adapted  to  the  require¬ 
ments  established  for  CISOs  by  the  various  CNET  and  CNTECHTRA  instructions. 

The  commands  have  established  these  offices,  and  training  development,  instruc 
tional  support,  and  training  evaluation  functions  are  performed  locally. 

To  say  that  the  CISO  concept,  in  its  broadest  sense,  is  not  working  would 
be  technically  incorrect.  For  the  most  part,  functions  required  by  higher 
authority  instructions  are  performed  by  someone  to  some  level  at  the  local 
command.  The  functions  are  not  necessarily  performed  by  CISO  staff,  however, 
nor,  in  the  usual  case,  are  they  performed  by  individuals  who  are  administra¬ 
tively  separate  from  the  instructional  departments.  The  issues  that  emerge 
are  not  so  much  compliance  with  the  instructions  but  the  manner  in  which 
compliance  has  been  made  and  whether  or  not  the  actions  taken  are  optimum 
and  indeed  conducive  to  good  quality  training. 

No  immediately  available  objective  criteria  or  standards  can  be  employed 
to  assess  the  adequacy  or  effectiveness  of  CISO  performance  in  terms  of 
direct  contributions  to  achieving  quality  training.  Consequently,  the  "ideal” 
CISO  configuration  could  not  be  objectively  determined  by  the  study.  At 
several  activities  visited,  however,  local  command  staff  expressed  convictions 
that  their  own  CISO  organization  was,  in  fact,  ideal.  And,  local  commanding 
officer/executive  officer  respondents  to  the  short  questionnaire  did  report 
that  they  were  both  generally  satisfied  with  the  performance  of  their  CISOs, 
and  that  the  offices  were  generally  effective.  In  the  absence  of  more 
objective  criteria,  however,  these  opinions  could  well  be  interpreted  to 
mean  that  the  local  CISO  has  been  organized  and  performs  in  ways  that  are 
satisfactory  to  the  local  command.  Stated  another  way,  the  local  command 
has  found  a  workable  solution  to  the  problem  of  accomplishing  a  wide  variety 
of  functions  within  the  confines  of  available  resources. 
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RESOURCE  AND  PERSONNEL  UTILIZATION  PR08LEMS.  Virtually  all  of  the  CISOs 
contacted  during  the  study  noted  that  they  lacked  sufficient  manpower  to 
perform  the  functions  expected  of  CISOs.  The  implication  is  that  many 
functions  nominally  designated  for  accomplishment  by  the  CISO  are  not  done 
there  because  of  the  absolute  lack  of  manpower.  Occasionally,  they  are  not 
done  because  the  assigned  staff  lack  the  necessary  skills  to  perform  the 
functions.  Thus,  the  indicated  solution  is  to  staff  "properly"  CISOs  and 
the  functions  will  thereafter  be  performed.  However,  this  is  a  deceptively 
simple  solution  in  the  CISO  case  and  one  that  requires  further  discussion. 

The  manpower  resource  problem  must  consider  not  only  numbers  of  individuals 
but  also  the  types  of  talent  required  for  effective  performance  of  CISO 
functions.  An  additional  problem  involves  properly  using  talent.  Specific 
personnel  resource  problems  and  related  factors  which  apparently  affect 
decisions  concerning  personnel  use  are  discussed  below. 

Lack  of  Skills.  The  specific  skills  required  to  perform  some  of  the  tasks 
assigned  to  CISOs  are  frequently  not  available  at  local  commands.  Effective 
performance  of  the  three  major  categories  of  functions  assigned  CISOs  (i.e., 
training  development,  training  evaluation,  and  instructional  support)  require 
specialized  skills--subject  matter  expertise  is  not  sufficient. 

Effective  training  development  requires  personnel  with  knowledge  of 
educational  technology.  They  must  also  be  able  to  apply  this  technology  to 
designing  and  developing  courses/modules  of  instruction.  Training  evaluation 
requires  a  different  set  of  skills  and  knowledges.  In  this  area,  personnel 
are  required  who  understand  evaluation  goals  and  can  apply  appropriate 
evaluation  procedures.  Accomplishing  instructional  support  functions  requires 
not  only  personnel  capabilities  in  the  above  areas  but  certain  communication 
skills  are  also  desirable  for  providing  training  or  conducting  workshops 
and  generally  assisting  others  to  accomplish  training  quality  functions. 

Many  of  the  skills  necessary  for  effectively  accomplishing  CISO  functions 
are  not  routinely  possessed  by  assigned  enlisted  personnel.  They  are  also, 
at  a  number  of  training  activities,  probably  not  possessed  in  the  depth 
required  by  the  assigned  education  or  training  specialists.  Given  the  lack 
of  the  specialized  skills  required  for  effectively  performing  CISO  functions, 
some  assigned  tasks  are  not  done  at  all  and  some  are  simply  not  done  well. 

Personnel  Utilization  Factors.  Currently,  there  is  a  general  lack  of  manpower 
at  local  training  activities  to  perform  all  of  the  functions  (not  just  CISO 
functions)  required  to  produce  graduates.  Resources  assigned  to  the  local 
school  are  under  the  direct  control  of  the  commanding  officer  to  use  as  his 
best  judgment  dictates  to  fulfill  the  assigned  training  mission.  Given  the 
current  shortages  of  personnel  to  perform  all  functions  at  training  activities, 
tne  local  commanding  officer  is  often  faced  with  the  prospect  of  eithe 
"robbing"  the  instructional  departments  to  staff  CISOs  or  "robbing"  CISOs 
to  perform  other  tasks.  In  many  instances,  local  commanders,  perhaps  for 
lack  of  other  alternatives  and  perhaps  for  other  reasons,  use  personnel 
designated  as  CISO  staff  to  perform  tasks  which  are  not  in  keeping  with  the 
instructions  governing  CISO  responsibilities.  For  example,  most  CIS  officers 
are  assigned  one  or  more  additional  duties  as,  for  example.  Legal  Services 
Officer,  Recreation  Officer,  Watch  Bill  Coordinator,  Training  or  Training 
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Support  Department  Heads,  United  Fund  Coordinators,  Educational  Services 
Officer,  Personnel  or  Student  Control  Officer.  It  is  difficult  in  many 
cases  to  decide  which  is  the  collateral  duty,  CIS  officer  or  the  other  assign¬ 
ments.  In  addition,  many  senior  enlisted  personnel  function  as  instructors 
and  also  perform  Cl  SO  duties.  Similarly,  assigned  education  specialists 
are  frequently  used  to  perform  general  administrative  and  sometimes  almost 
clerical  duties  (e.g.,  accounting,  administrative  record  keeping,  manpower, 
student  scheduling,  budgeting). 

Factors  that  appear  to  operate  (singly  or  in  combination)  to  influence 
decisions  for  personnel  utilization  include: 

•  production  pressures  leading  to  preoccupation  with  the  production 
process 

•  lack  of  understanding  of  the  need  for  and  nature  and  purpose  of 
functions  designed  to  promote  quality  in  training 

•  implicit  beliefs  that  only  instructors  can  achieve  high  quality 
training  coupled  with  unfavorable  appraisals  of  CISO  staff  capa¬ 
bilities. 

The  pressure  to  produce  graduates  is  one  factor  that  probably  affects 
the  degree  of  attention  given  to  training  quality  functions  and,  consequently, 
personnel  assignments.  This  pressure,  undoubtedly,  contributes  to  a  reluc¬ 
tance  to  expend  time  and  resources  on  performing  functions  that  are  not 
considered  as  important  as  "getting  graduates  out."  Some  balance  between 
meeting  production  requirements  and  ensuring  training  quality  is  obviously 
a  desirable  goal. 

Discussions  with  local  CISO  staffs  led  to  the  conclusion  that  in  many 
instances  there  may  be  less  than  adequate  understanding  on  the  part  of  local 
command  personnel  (including  some  CISO  personnel)  of  the  nature  and  purpose 
of  functions  and  operations  required  to  assure  local  training  quality.  In 
this  area,  short  courses  covering  CNET  purposes,  goals,  and  procedures  for 
areas  such  as  Instructional  Systems  Development  and  evaluation  techniques 
would  be  beneficial  for  local  command  personnel  (especially  all  officers). 

A  number  of  local  CISO  staff  (mostly  education  specialists)  also  noted 
during  visits  that  some  local  commanding  officers  do  not  emphasize  to  their 
commands  that  the  functions  required  by  the  CISO  instruction  should  be  accom¬ 
plished  by  the  CISO.  CISO-type  functions  that  are  performed  by  the  training 
departments  may  or  may  not  be  subject  to  CISO  review  or  concurrence.  In 
many  instances,  CISO  staff  have  very  limited,  almost  negligible,  cognizance 
over  the  work  or  end  products  of  the  training  departments.  With  respect  to 
performing  curriculum  development  functions,  this  may  reflect  an  attitude 
that  only  subject  matter  expert  (SME)  instructors  are  qualified  to  develop 
instruction.  It  may  also  reflect  a  lack  of  understanding  of  what  is  required 
for  systematic  training  development  (e.g.,  ISO)  or  what  an  individual  trained 
in  educational  technology  can  contribute  to  curriculum  development. 
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An  additional  consideration  is  the  current  shortage  of  SMEs  for  instruc¬ 
tional  duties.  Local  school  commanders  are  probably  reluctant  to  assign 
staff  to  CISOs  (or  IPDCs)  exclusively  for  duty  as  curriculum  developers. 
Performing  curriculum  development  functions  in  the  training  department  where 
SMEs  can  also  double  as  instructors  would  be  desirable  at  some  activites. 
However,  this  practice  should  not  arbitrarily  exclude  CISO  personnel  from 
that  activity. 

A  more  subtle  problem  that  affects  local  CISO  ability  to  function  effec¬ 
tively  relates  to  the  interface  between  CISO  and  training  department  staffs. 

A  number  of  CISO  personnel  reported  that  training  departments  tend  to  ignore 
their  existence.  Consequently,  they  do  not  "invite"  the  CISO  to  participate 
in  performing  functions  nominally  assigned  to  CISO  by  higher  command  instruc¬ 
tions.  The  training  departments  consider  that  these  functions  should  be 
done  by  the  training  departments.  Such  functions  include,  most  notably, 
curriculum  development  and  test  construction.  At  a  number  of  activities 
visited,  CISOs  are  excluded  from  meaningful  participation  in  the  curriculum 
development  process.  Even  course  outlines  developed  locally  may  not  be 
reviewed  by  CISOs  but  go  directly  from  the  training  department  to  CNTECHTRA 
for  approval.  Many  CISOs  state  that  they  assist  the  training  departments 
on  an  "as  requested"  basis,  out  their  services  may  not  often  be  requested. 

This  may  reflect  an  attitude  that  CISOs  can  not  help  because  they  are  not 
usually  subject  matter  experts.  In  smaller  schools,  training  departments 
tend  to  make  better  use  of  CISO  staff,  perhaps  because  of  the  more  apparent 
need  to  use  all  of  the  help  available. 

An  additional  problem  may  be  the  perceived  role  by  military  personnel 
of  the  civilian  education  specialists  in  the  schools.  Most  CISOs  are  staffed 
with  education  specialists  who  most  often  sit  in  a  higher  level  of  command 
than  do  the  departmental  instructors.  Thus,  the  reluctance  to  use  CISO 
services  may  really  reflect  a  reltctance  to  work  with  civilian  education 
specialists  who  are  in  a  position  to  review/criticize  their  efforts.  And, 
instructors  who  will  subsequently  conduct  the  training  may  simply  feel  that 
they  should  have  exclusive  cognizence  in  its  development. 

INSTRUCTION  LANGUAGE.  One  final  factor  that  probably  affects  what  local 
CISOs  do  is  the  language  of  the  CfET  and  CNTECHTRA  CISO  instructions.  The 
functions  identified  for  CISO  performance  are  preceded  by  a  variety  of  action 
verbs  (e.g.,  administer,  coordinate,  standardize,  monitor)  whose  meanings 
are  not  always  clear  in  the  context  provided  (see  appendix  A).  Thus,  in 
many  cases,  local  commands  may  not  have  a  clear  understanding  of  how  higher 
authority  expects  CISOs  to  function. 

CONCLUSIONS  AND  COMMENTS 

Information  obtained  from  the  survey  of  CISOs  leads  to  the  general 
conclusion  that  the  CISO  concept  has  not  been  well  implemented  within  the 
'  T'/EDTRACOM.  A  major  reason  for  this  inadequate  implementation  may  be  the  _ 
general  lack  of  manpower  at  local  commands  for  performing  all  of  the  functions 
required  to  fulfill  the  assigned  training  mission.  The  lack  of  manpower 
results  in  dual  personnel  assignments  to  both  training  and  CISO  jobs.  Thus, 
the  required  administrative  separation  of  instructional  departments  and  the 
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CISO  has  not  been  achieved.  In  addition,  functions  designated  for  CISO 
accomplishment  often  may  not  be  performed  at  all,  or  may  be  performed 
inadequately.  For  CISOs,  there  are  the  additional  problems  of  lack  of 
specialized  skills  required  for  effective  performance  of  expected  func¬ 
tions,  improper  utilization  of  personnel,  and  proprietary  issues  involving 
roles  and  interrelationships  of  local  personnel. 

The  problems  of  CISOs  are  complexly  intertwined  and  no  simple  solutions 
to  them  are  apparent.  It  seems  clear,  however,  that  simply  adding  resources 
to  the  local  commands  to  perform  CISO  functions  will  not  solve  all  the  problems, 
given  the  local  commanding  officer's  prerogatives  and  perceptions  concerning 
how  best  to  utilize  his  resources  to  accomplish  his  assigned  training  mission. 

At  the  very  least,  some  tightening  of  controls  over  and/or  greater  attention 
to  local  training  is  needed  to  ensure  the  quality  of  that  training.  At 
present,  there  seems  to  be  no  effective  system  whereby  local  commanding 
officers  are  held  directly  accountable  for  the  quality  of  the  training  given 
by  their  activity.  Some  system  of  accountability  should  be  established  to 
ensure  that  certain  specialized  personnel  resources  (e.g.,  education  special¬ 
ists)  are  used  to  perform  tasks  consonant  with  their  training  and  experience. 
Similarly,  some  system  is  needed  for  ensuring  that  required  quality  assurance 
functions  are,  in  fact,  being  performed  locally  and  for  checking  overall 
training  quality. 

The  basic  concept  of  an  office  within  a  local  training  activity  charged 
with  responsibility  for  assuring  training  quality  is  sound.  However,  changes 
to  the  current  implementation  of  this  concept  are  clearly  desirable. 


53/54 


TAEG  Report  No.  110 


SECTION  VI 

ALTERNATIVES  FOR  TRAINING  QUALITY  ASSURANCE 

Information  obtained  during  the  study  led  to  the  conclusion  that,  in 
general,  the  local  CISO  concept  as  a  means  of  assuring  training  quality  has 
not  been  well -implemented.  Consequently,  as  required  by  the  basic  CNET  task¬ 
ing,  a  number  of  alternatives  to  this  concept  were  developed  and  assessed. 

These  alternatives  (options)  were  aimed  at  alleviating  or  overcoming  problems 
that  affect  CISO  ability  to  perform  effectively.  In  developing  options. 

Army  and  Air  Force  training  quality  assurance  (QA)  practices  were  appropri¬ 
ately  considered.  (These  practices  are  summarized  in  appendix  C.) 

Initially,  two  "sets"  of  options  were  developed:  nondisruptive  and 
disruptive.  The  nondisruptive  options  feature  minimal  to  moderate  change 
to  existing  command  organizations  and  the  current  distribution  of  training 
QA  functions.  These  options  are  based  on  consideration  for  current  character¬ 
istics  of  the  NAVEDTRACOM;  they  propose  no  major  organizational  (or  func¬ 
tional)  changes.  The  disruptive  options  would  require  substantial  changes 
to  the  current  NAVEDTRACOM  organization  and  to  the  distribution  of  quality 
assurance  functions.  These  options  address  more  the  aspects  of  a  command 
training  management  system  than  simply  how  to  accomplish  QA  functions  at 
the  school  level.  Each  set  of  options  is  more  fully  discussed  below. 

NONDISRUPTIVE  OPTIONS 

Table  12  lists  a  number  of  nondisruptive  options  that  could  be  implemented 
to  accomplish  training  quality  assurance  functions. 

•  Option  1,  Maintain  the  status  quo  for  all  CISOs.  This  is  the 
easiest  option  to  implement.  However,  this  option  is  undesirable, 
since  it  would  ignore  the  problems  associated  with  the  CISO  concept. 

•  Option  2,  No  organizational  change;  provide  additional  manpower. 

This  option  is  also  undesirable  since  many  problems  identified 
previously  would  remain.  However,  additional  manpower  would  be 
provided  to  accomplish  assigned  duties  and  tasks.  Yet,  this  option 
presents  the  additional  problem  of  deciding  what  manpower  would 

be  required.  Some  CISOs  could  be  adequately  staffed  by  the  addition 
of  one  billet  while  others  could  require  20  to  30  additional  billets. 
If  this  option  were  selected,  a  study  would  be  required  to  determine 
the  appropriate  manpower  for  each  CISO. 

•  Option  3,  No  organizational  change;  institute  an  accountability 
system  with  more  centralized  control.  Central  management  to  assign 
priority  of  task  to  be  completed.  Local  training  personnel  in 

the  current  system  may  have  good  knowledge  of  needed  curriculum 
changes  from  feedback  received  informally  from  the  fleet.  However, 
these  local  personnel  often  do  not  understand  who  actually  has 
control  over  a  course  curriculum  or  who  has  the  authority  to  approve 
changes.  The  lack  of  clear  lines  of  authority  causes  delay  in 
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TABLE  12.  NONDISRUPTIVE  OPTIONS 


1.  Maintain  the  status  quc  for  all  CISOs. 


2.  No  organizational  chance;  provide  additional  manpower. 


3.  No  organizational  change;  institute  an  "accountability" 

system  with  more  centralized  control.  Central  management  to 
assign  priority  of  task  to  be  completed. 


4.  Abolish  CISO;  put  personnel  in  schoolhouse. 


5.  Split  functions  and  assignments 


Course  Development 

Evaluation 

a.  Contract 

CNTECHTRA 

b.  IPDC 

CNET 

c.  School 

School 

d.  CISO 

CISO 

e.  Combination  of  a/b/c/d 

Combination  of  a/b/c/d 

6.  Organize  CISOs  based  upon  needs  of  different  commands. 
Build  a  flexible  organizational  system. 
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approval  of  curriculum.  It  also  contributes  to  lack  of  uniformity 
across  training  activities.  Centralizing  control  would  reduce 
confusion  over  authority  and  increase  uniformity.  However,  such 
centralized  control  often  fails  to  take  into  account  special  local 
problems  and  is  typically  less  flexible. 

•  Option  4,  Abolish  Cl SO;  put  personnel  in  schoolhouse.  This  option 
abolishes  the  CISO  but  it  does  not  abolish  the  need  to  perform 
various  functions.  If  this  option  were  adopted,  there  might  very 
well  be  a  reduction  in  attention  to  quality  assurance  on  the  part 
of  local  training  activities. 

•  Option  5,  Split  functions  and  assignments.  This  option  comprises 
several  alternatives  for  course  development  and  evaluation.  It 
provides  maximum  flexibility  for  accomplishing  any  task  since  it 
could  be  assigned  to  the  school,  a  contractor,  IPDC,  CISO,  CNTECHTRA, 
CNET  or  any  combination  thereof.  The  problems  are  funding,  coordina¬ 
tion  among  many  different  activities,  and  accountability.  However, 
some  combinations  under  this  option  are  desirable  and  may  be  neces¬ 
sary  in  the  near  future.  The  most  likely  combination  that  will 
emerge  is  having  contractors  assist  in  course  development.  Over 

500  instructor  billets  recently  have  been  contracted  out,3  setting 
a  precedent  in  using  civilian  instructors  in  Navy  schools.  If 
this  trend  continues,  it  may  be  necessary  to  arrange  for  a  portion 
of  civilian  instructors'  time  to  be  devoted  to  curriculum  design/ 
redesign. 

•  Option  6,  Organize  CISOs  based  upon  needs  of  different  commands. 

Build  a  flexible  organizational  system.  This  option  reflects  the 
current  status  of  CISOs.  This  study  found  that  the  CISOs  surveyed 
varied  considerably  in  their  operation.  This  may  appear  reasonable 
since  each  CISO  operates  within  a  unique  context  and  flexibility 

is  useful  to  accomplish  local  responsibilities.  However,  unlimited 
flexibility  is  undesirable.  Some  local  commands  do  not  currently 
use  their  personnel  appropriately  (e.g.,  some  education  specialists 
perform  administrative  duties).  Section  VII  of  this  report  presents 
the  functions  that  should  be  performed  at  all  training  activities 
to  assure  the  quality  of  training. 

DISRUPTIVE  OPTIONS 

Disruptive  options  considered  as  alternatives  to  the  CISO  concept  are 
shown  in  table  13.  Each  option  presented  is  discussed  below. 


3CNET  msg  (NOTAL)  310013Z  May  1978. 
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TABLE  13.  DISRUPTIVE  OPTIONS 

1.  Model  CISOs  based  upon  Air  Force  concept 

•  Responsibility  for  courses  designated  to  school 
»  Single  point  manager  for  a  course 

•  Complete  control  of  course  by  TPC. 

2.  Assign  CISO  personnel  to  IPD  Centers. 

3.  Assign  IPD  Center  personnel  to  CISO/schools  on  loan  basis. 

4.  Visiting  team  concept  for  evaluation  and/or  curriculum 
development. 

Drawn  from: 

•  IPDC 

•  CNTECHTRA 

•  SMEs. 

5.  Use  contractors  for  al  course  development/evaluation. 
Controlled  from: 

•  Central  location 

•  CISOs  (redistributed). 
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Option  1,  Model  the  CISO  based  upon  the  Air  Force  concept.  Single 
point  management  for  courses  is  a  central  feature  of  Air  Force 
training.  This  concept  locates  technical  and  administrative  con¬ 
trol  of  courses  at  the  school  level.  TPCs  could  be  assigned  to 
schools  to  perform  the  same  role  as  the  course  manager  in  an  Air 
Force  School  (see  appendix  C).  Placing  the  TPC  in  the  school 
would  in  some  cases  provide  an  educational  technologist  to  the 
school  as  well  as  locate  managerial  control  of  the  course  at  the 
local  level.  One  problem  of  implementing  this  concept  is  that 
additional  TPCs  would  be  required  since  there  are  over  4,000 
courses  in  the  Navy. 


Option  2,  Assign  CISO  personnel  to  IPDCs.  This  option  is  feasible 
for  schools  in  San  Diego,  Great  Lakes,  Memphis,  and  Pensacola, 
where  there  are  IPDCs.  Potential  problems  with  this  concept, 
however,  are:  (a)  remote  location  of  personnel  from  courses  that 


need  revision,  (b)  reduction  of  personnel  at  schoolhouses,  (c) 


"not  developed  here"  syndrome. 


•  Option  3,  Assign  IPDC  personnel  to  CISO/schools  on  loan  basis. 

This  option  would  provide  additional,  specialized  manpower  to 
schools  for  short  durations.  However,  under  their  current  work¬ 
loads  the  IPDCs  do  not  have  adequate  personnel  to  perform  their 
presently  assigned  t'sks. 

•  Option  4,  Visiting  team  for  evaluation  and/or  curriculum  development. 
A  team  of  NAVEDTRACOM  personnel  with  specific  talents  and  expertise 
could  be  assembled  arid  sent  to  local  training  activities  to  train 
local  personnel  and  to  provide  direct  assistance.  CNTECHTRA  (Code 
N63)  currently  provides  some  support  of  this  type  to  CISOs.  Negative 
aspects  of  this  concept  include  the  need  for  extensive  travel 
funds,  the  reluctance  of  many  people  to  travel  for  extended  periods 
of  time,  and  the  loss  of  these  personnel  at  their  home  duty  station 
for  long  periods  of  time.  However,  this  option  does  offer  a  promis¬ 
ing  short  range  solution  to  the  manpower  shortage  at  the  CISOs. 

•  Option  5,  Use  contractors  for  all  course  development/evaluation. 
Another  promising  option  to  alleviate  the  shortage  of  manpower  is 
to  use  contractors  for  some  quality  assurance  functions.  The 
positive  aspects  of  using  contractors  include  expertise  and  suffi¬ 
cient  manpower.  The  negative  aspects  include  cost,  monitoring 
problems,  and  lack  of  responsiveness  to  the  school.  Currently, 

the  NAVEDTRACOM  is  establishing  an  organization  to  provide  contract¬ 
ing  assistance  to  its  functional  commands.  This  organization, 
the  Commercial  and  Industrial  Type  Activity  (CITA),  could  be 
requested  to  analyze  CISO  functions  to  assess  the  cost  effective¬ 
ness  of  contracting  some  of  them  out. 

Having  considered  various  options  for  training  quality  assurance,  however, 
the  TAEG  concluded  that  the  CISO  concept  should  be  retained.  Consequently, 
a  proposed  charter  for  the  future  operation  of  CISOs  was  written.  This 
charter  incorporates  aspects  of  several  of  the  options  discussed  above; 
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however,  it  assigns 
the  local  command, 
cf  this  report. 


primary  responsibility  for  the  quality  of  training  to 
The  proposed  charter  is  presented  in  the  next  section 
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SECTION  VII 

A  PROPOSED  CHARTER  FOR  CISOs 

This  section  presents  a  brief  rationale  for  retention  of  the  CISO  con¬ 
cept  and  a  proposed  charter  for  the  future  operation(s)  and  organization  of 
these  offices. 

RATIONALE  FOR  APPROACH  CHOSEN 

Intensive  discussions  held  within  TAEG,  with  CISO  staffs  throughout 
the  NAVEDTRACOM,  and  with  other  services'  staffs  led  to  the  conclusion  that 
operations  to  enhance  training  quality  must  occur  at  the  local  schoolhouse 
level  and  that  responsibility  for  performing  necessary  QA  functions  should 
be  vested  at  that  level. 

The  basic  notion  underlying  the  CNET  and  CNTECHTRA  instructions  that 
established  CISOs  within  the  NAVEDTRACOM  is  that  certain  functions  should 
be  performed  at  the  local  schoolhouse  level  to  assure  Nawy  training  quality. 

In  this  sense,  the  instructions  are  attention-directing.  They  list  functions 
that  should  be  performed  and  provide  a  mechanism  in  the  CISO  concept  for 
their  accomplishment.  Even  though  resource  and  other  constraints  may  operate 
to  limit  the  degree  to  which  CISOs  can  accomplish  QA  functions,  the  existence 
of  the  concept  provides  at  least  a  minimum  guarantee  that  certain  features 
of  local  training  will  receive  more  attention  than  they  would  have  if  the 
concept  had  not  been  established.  In  addition,  the  instructions  recognize 
that  systematically  applying  modern  educational  technology  in  designing  and 
developing  instruction,  continuously  evaluating  the  instructional  system, 
and  correcting  deficiencies  are  necessary  for  establishng  and  maintaining  a 
high  level  of  training  quality.  Thus,  there  are  a  number  of  positive  fea¬ 
tures  and  benefits  for  training  associated  with  the  notion  of  a  local  quality 
assurance  office  or  CISO.  However,  certain  adjustments  to  the  current  imple¬ 
mentation  of  the  CISO  concept  are  indicated  to  promote  the  cause  of  training 
quality. 

RECOMMENDED  CHARTER 

The  remainder  of  this  section  presents  recommendations  for  the  future 
accomplishment  of  training  quality  assurance  functions. ^  The  recommenda¬ 
tions  are  given  in  the  form  of  a  "charter"  for  a  quality  assurance  office 
(or  CISO)  to  operate  within  the  local  school.  The  functions  considered 
necessary  for  assuring  training  quality  are  presented  first.  Where  necessary, 
the  purpose  and  meaning  of  functions  are  defined.  A  recommended  level  of 
involvement  for  a  local  QA  office  is  also  given.  Level  of  involvement 


4The  term  "quality  assurance"  should  be  interpreted  in  a  broad  sense  to 
include  consideration  f<r  all  efforts  required  to  establish,  implement,  and 
maintain  a  high  level  of  quality  in  training.  It  includes  concern  for  the 
design  and  development  of  curricula  and  training  aids  and  materials,  instruc¬ 
tional  support,  and  training  evaluation. 
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considerations  are  based  principally  on  the  recognition  that  the  greatest 
potential  contribution  to  training  quality  fror.  individuals  normally  connected 
with  CISOs  is  in  the  area  of  educational  technology  and  not  in  subject  matter 
expertise.  Considerations  for  organization  anc  staffing  of  CISOs  are  presented 
after  the  suggested  functions. 

The  TAEG  endorses  the  notion  that  the  CISO  should  continue  to  be  an 
integral  part  of  the  activity  which  it  serves  and  not  an  agent  of  higher 
authority.  However,  based  on  findings  of  the  study,  it  appears  desirable 
to  strengthen  the  interface  between  local  CISOs  and  CNTECHTRA  (especially 
CNTECHTRA  Code  N63)  for  specific  purposes  designed  to  assist  in  achieving 
quality  training.  Consequently,  specific  recommendations  in  this  area  are 
also  presented.  It  also  became  clear  during  the  study  that  certain  minimum 
enforcement  mechanisms  would  be  desirable  in  the  future  to  ensure  that  neces¬ 
sary  training  quality  assurance  functions  are  accomplished  at  the  schools, 
and  that  the  talents  of  individuals,  both  SMEs  and  education  and  training 
specialists,  are  appropriately  used  to  meet  the  common  goal  of  high  quality 
training. 

The  discussions  presented  below  focus  on  each  element  comprising  the 
proposed  CISO  charter,  specifically: 

•  quality  assurance  functions 

•  CISO  organization  and  staffing 

•  higher  command  interfaces 

•  accountability  mechanisms. 

QUALITY  ASSURANCE  FUNCTIONS.  As  mentioned  previously,  an  overall  list  of 
functions  currently  considered  appropriate  for  CISOs  to  perform  was  compiled 
from  applicable  instructions  and  from  inputs  received  from  the  CISO  survey 
(visits  and  questionnaires)  and  from  CNTECHTRA  staff.  This  overall  list  is 
presented  in  appendix  A.  It  served  as  a  principal  basis  for  developing 
functions  to  be  performed  to  promote  local  training  quality  in  the  future. 

All  functions  listed  were  carefully  examined  and  deletions  were  made  from 
the  list.  The  functions  eliminated  were  those  which,  in  the  judgment  of 
TAEG,  are  unnecessary  or  inappropriate  for  a  local  training  quality  assurance 
operation.  This  judgment  was  aided  b.'  inputs  received  from  CISOs,  CNTECHTRA 
staff,  and  Army  and  Air  Force  personnel.  The  resulting  list  of  functions 
are  those  which  are  considered  approp-late  for  assignment  to  a  CISO  or  CISO- 
like  organization  functioning  at  the  local  schoolhouse  level. 

It  is  believed  that  the  functions  retained  best  reflect  the  spirit  of 
the  basic  CNET  and  CNTECHTRA  instructions;  i.e.,  training  quality  assurance. 
Effective  accomplishment  of  these  functions  requires  applying  educational 
technology  and  evaluation  skills.  Only  a  limited  number  of  administrative 
functions  have  been  retained  for  consideration  of  performance  by  a  CISO  or 
C I  SO- 1  ike  organization.  These  are  functions  that  are  either  logical  out¬ 
growths  of  technical  functions  Identified  for  CISO  performance  or  that  cut 
a.ross  departmental  lines  and  thus  make  a  CISO-like  organization  a  logical 


62 


TAEG  Report  No.  110 


candidate  for  their  accomplishment.  Functions  that  serve  only  one  department 
or  that  make  the  CISO  little  more  than  an  administrative  unit  for  a  training 
department  were  eliminated  from  consideration.  Similarly,  those  functions 
that  appeared  to  involve  only  material  storage  or  that  were  essentially 
clerical  were  eliminated.  In  short,  the  functions  advanced  for  accomplish¬ 
ment  by  CISOs  are  those  based  on,  and  requiring,  proper  employment  of  techni¬ 
cal  talent  in  the  interest  of  training  quality  assurance.  The  functions 
eliminated  from  the  basic  list  (appendix  A)  should  probably  continue  to  be 
performed  at  local  levels;  however,  they  are  considered  inappropriate  for 
an  organization  that  should  be  directly  concerned  with  and  responsible  for 
training  quality. 

The  functions  remaining  after  the  elimination  process  described  above 
were  grouped  into  five  major  categories.  The  five  major  categories  listed 
in  order  of  importance  for  assuring  training  quality  are: 

•  Curriculum  Design/Redesign 

•  Evaluation 

•  Instructional  Support 

•  Administrative 

•  Special  Training  Support. 

Each  category  requires,  for  the  most  part,  different  sets  of  technical  skills. 
The  first  three  sets  of  functions  represent  a  minimum  list  of  efforts  that 
should  be  accomplished  at,  or  for,  al^  schools  to  assure  the  quality  of  a 
school's  training.  These  functions  should  be  performed  by  individuals  who 
possess  appropriate  educational  technology  and  evaluation  skills.  Suggestions 
for  augmenting  local  staffs  are  provided  on  page  72  under  "Organization  and 
Staffing  Considerations." 

In  developing  the  recommendations  presented  in  this  section,  the  reali¬ 
ties  of  the  local  school  context  have  been  considered  and  a  concerted  attempt 
has  been  made  to  achieve  a  balance  between  resources  needed  and  resources 
currently  (or  likely  to  be)  available.  Although  the  functions  identified 
should  be  performed  at  all  activities,  a  considerable  degree  of  flexibility 
has  been  retained  to  permit  local  commanding  officers  to  decide  how  best  to 
accomplish  the  functions  within  his/her  own  activity. 

Figure  2  displays  schematically  the  functional  areas  of  responsibility 
recommended  for  CISOs.  Beneath  each  of  the  five  major  categories  of  func¬ 
tions,  more  specific  functions,  and  recommended  involvement  are  displayed. 

Each  group  of  recomnended  QA  functions  and  subfunctions  is  discussed 
separately  below.  Note  that  the  language  in  which  the  functions  were  origin¬ 
ally  expressed  (see  appendix  A)  has  not  always  been  retained.  Where  necessary 
for  clarification  of  intent,  functions  are  defined.  In  addition  to  listing 
the  functions,  a  recomnended  nature  and  level  of  involvement  in  accomplishing 
functions  by  a  local  CISO  is  given.  Recommendations  for  level  and  nature 
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of  involvement  reflect  the  opinions  of  various  NAVEDTRACOM  and  TAEG  personnel 
and  practices  of  other  military  services. 

Curriculum  Design/Redesign  Functions.  Suggestions  concerning  how  local 
CISOs  should  be  involved  in  curriculum  design/redesign  functions  are  given 
in  table  14.  The  functions  that  must  be  performed  are  specified  in  NAVEDTRA 
110A.  The  numbers  in  the  cells  of  the  table  (keyed  to  the  action  verbs  at 
the  bottom  of  the  table)  denote  the  recommended  nature  of  CISO  involvement 
in  each  function.  Recommendations  for  CISO  involvement  in  functions  are 
shown  for  four  different  conditions  under  which  curriculum  development  work 
may  occur: 


•  local  development  in  which  the  local  school  is  directly  responsible 
for  the  effort 

•  IPDC  responsibility  for  the  development  work 

•  contract  development  where  the  local  school  is  cognizant  of  the 
effort 

•  contract  development  where  some  other  activity  (e.g.,  IPDC, 
CNTECHTRA)  is  cognizant  of  the  effort. 

Recommendations  concerning  CISO  involvement  in  curriculum  design/ 
redesign  functions  represent  the  collective  opinions  of  various  NAVEDTRACOM 
personnel.  Individuals  providing  the  rankings  were  CNET  and  CNTECHTRA  staff 
personnel  highly  knowledgeable  in  ISD  principles  and  procedures,  TPCs,  local 
CISO  staff,  and  local  training  activity  commanding  officers.  Thus,  a  cross- 
section  of  command  opinion  is  reflected  in  the  composite  recommendations 
given  in  table  14.  Using  the  action  verbs  listed  at  the  bottom  of  table 
14,  15  individuals  independently  recorded  their  first,  second,  third,  etc., 
choices  concerning  how  CISOs  should  be  involved  in  the  functions.  The  assump¬ 
tions  under  which  the  rankings  were  made  are: 

•  ISD  procedures  must  be  followed  in  curriculum  design/redesign 

•  the  local  CISO  representative  is  an  education  specialist  trained 
in  ISD  procedures 

•  the  local  CISO  representative  is  not  a  fully  qualified  subject 
matter  expert. 

The  TAEG  recommends  that  the  entries  in  the  table  be  considered  as  minimum 
ways  in  which  CISO  staff  be  involved  in  the  future  in  curriculum  design/ 
redesign  functions. 

Additional  specific  recommendations  for  CISO  involvement  in  this  area 
are  that  the  CISO: 


advise/assist  curriculum  developers  with  the  integration  of 
special  areas  such  as  PQS,  3M,  safety 
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•  coordinate  interservice  training  requirements  relative  to 
curricula. 

Further,  the  CISO  should  continue  to  review  and  approve  all  locally 
produced  training  publications.  This  office  should  also  be  involved  in 
periodic  reviews  of  all  courses  (i.e.,  course  materials,  training  aids  and 
devices,  objectives)  and  curriculum  control  documents  to  ensure  their  currency 
and  accuracy. 

Evaluation  Functions.  Evaluation  functions  with  which  CISO  personnel  might 
be  effectively  involved  are  presented  below  in  four  broad  groups:  student 
testing,  attrition  and  setbacks,  external  evaluation,  and  internal  evaluation. 
The  discussion  of  these  functions  includes  definitions,  rationale  for  their 
desirability,  and  appropriate  levels  of  involvement  for  CISO  personnel. 

Student  Testing.  The  principal  role  of  CISO  personnel  in  student  testing 
programs  should  be  one  of  providing  professional  advice  to  instructional 
personnel  concerning  constructing  and  revising  tests.  This  would  include 
consultation  concerning  format,  administration,  and  interpretation  of  test 
results.  Also,  the  evaluation  of  the  psychometric  properties  of  tests  (e.g., 
reliability,  validity,  and  discrimination)  is  highly  desirable.  Test  evalua¬ 
tion  implies  collecting  test  results  and  maintaining  test  item  data  banks. 
However,  this  record  keeping  is  essentially  a  clerical  or  administrative 
function,  and  it  is  not  essential  that  the  individual  or  group  collecting 
the  data  actually  reside  in  the  CISO.  Such  assignment  of  clerical  or  adminis¬ 
trative  functions  is  probably  best  left  to  the  commanding  officer,  with  the 
understanding  that  the  responsible  CISO  personnel  should  determine  the  appro¬ 
priate  format  and  content  of  the  test  item  data  bank. 

Attrition  and  Setbacks.  CISO  personnel  can  usefully  serve  as  professional 
advisors  concerning  academic  attrition  and  setback  problems.  As  advisors, 

CISO  personnel  would  design  studies  to  determine  causes  and  correlates  of 
attrition  and  setbacks.  On  the  basis  of  those  studies,  they  would  further 
recommend  plans  for  dealing  with  the  problems.  Such  actions  should  be  contin¬ 
gent  on  the  rate  (or  trends  in  rate)  of  attrition  and  setback.  For  example, 
rates  of  attrition  or  setback  approaching  or  exceeding  those  prescribed  by 
higher  authority  would  serve  as  an  indicator  that  studies  concerning  the 
problems  should  be  implemented.  Further,  if  trends  indicate  rising  rates 
of  attrition  or  setback,  action  might  be  taken  before  a  specified  rate  is 
reached.  It  should  be  noted  th^t  a  zero  attrition  or  setback  rate  is  not 
only  unlikely  but  m v.y  be  undesirable,  since  it  might  indicate  that  standards 
are  too  low.  The  preferred  methods  for  dealing  with  attrition  or  setbacks 
do  not  typically  include  lowering  of  standards. 

External  Evaluation.  The  actual  conduct  of  external  evaluation  is  typically 
the  province  of  activities  external  to  the  local  school.  However,  CISO 
personnel  can  make  valuable  professional  contributions  in  this  area  by  serving 
as  the  technical  liaison  between  the  schoolhouse  and  the  evaluation  activities 
of  higher  authority.  The  major  involvements  of  CISO  personnel  with  external 
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evaluation  should  be  to  advise  and  assist  instructional  personnel  with  the 
preparation  of  NAVEDTRACOM  TAS  Level  II  questionnaire  items,  to  interpret 
findings  from  external  evaluation  as  they  are  received  from  higher  authority, 
and  to  advise  and  assist  schoolhouse  personnel  concerning  appropriate 
responses  to  any  weakness(es)  indicated  by  the  evaluation.  This  level  of 
involvement  will  allow  the  local  command  to  make  maximum  use  both  of  the 
professional  capabilities  of  CISO  personnel  and  of  the  feedback  obtained 
from  a  formal  external  evaluation.  Such  external  evaluation,  standardized 
as  it  is  across  numerous  local  training  commands,  may  require  some  local 
interpretation  before  it  is  directly  applicable  at  the  local  level.  Further, 
the  mere  indication  of  a  problem,  necessary  though  it  is,  is  not  adequate 
for  the  solution  of  the  problem.  Therefore,  CISO  personnel,  in  conjunction 
with  others  at  the  local  command,  will  need  to  transform  that  feedback  into 
a  plan  of  corrective  action. 

Internal  Evaluation.  Evaluating  the  process  or  outcome  of  instruction  within 
the  schoolhouse  constitutes  internal  evaluation.  Internal  evaluation  involves 
numerous  steps  or  actions  and  comprises  the  major  portion  of  evaluation 
activities  recommended  for  CISO  personnel.  The  CISO  should  serve  as  an  arm 
of  the  commanding  or  executive  officer  for  internal  quality  control  and 
quality  assurance.  With  the  exception  of  specialized  technical  knowledge 
relevant  to  particular  ratings  in  the  Navy,  the  CISO  should  have  the  neces¬ 
sary  expertise  and  authority  to  plan  and  conduct  all  aspects  of  internal 
evaluation,  from  the  design  of  evaluation  programs  to  the  diagnosis  and 
prescriptive  remedies  delivered  to  the  executive  or  commanding  officer  con¬ 
cerning  all  aspects  of  the  training  mission  of  the  local  command.  Internal 
evaluation  should  be  a  continuous  process. 

Designing  and  executing  an  overall,  comprehensive  internal  evaluation 
program  is  complex  and  many  decisions  must  be  left  to  the  professional  discre¬ 
tion  of  local  CISO  personnel.  Although  there  will  be  certain  problems  common 
to  all  training  activities  and  certain  evaluation  techniques  of  value  to 
all  training  activities,  there  will  be  enough  variation  among  schoolhouses 
to  preclude  attempts  to  prescribe  standard  internal  evaluation  programs 
that  will  meet  the  needs  of  each  school.  Professional  personnel  in  the 
CISO,  in  consultation  with  instructional  personnel,  should  design  internal 
evaluation  programs  and  procedures  that  are  responsive  to  local  problems 
and  needs.  Results  from  formal  external  evaluations,  when  available,  should 
be  appropriately  used  by  local  training  activities  to  determine  desirable 
changes  in  internal  evaluation  programs. 

Personnel  returning  from  fleet  duty  to  the  schoolhouse  constitute  a 
source  of  training  feedback  information  that  can  also  be  used  for  internal 
evaluation  purposes.  A  detailed  structured  interview  procedure  is  available 
for  obtaining  information  from  this  source  (Hall  and  Hughes,  1980).  The 
CISO  should  have  the  necessary  professional  and  technical  skills  to  conduct 
all  internal  evaluation  efforts  for  the  local  command  and  to  compare,  combine, 
and  integrate  information  from  the  various  sources. 

The  majority  of  internal  evaluation  functions  should  be  handled  by 
CISO  personnel,  the  exception  being  those  functions  that  require  specific 
technical  expertise  in  various  Navy  ratings.  In  those  cases,  CISO  personnel 
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should  work  with  local  subject  matter  experts  and/or  with  TPCs  and  Technical 
Audit  teams  as  necessary.  CISO  personnel  should  have  cognizance  over  internal 
feedback  functions  that  require  applying  professional  educational  and  psycho¬ 
metric  skills.  These  functions  might  include,  for  example,  the  following: 

•  designing  internal  feedback  instruments,  such  as  student  critique 
forms  and  instructor  evaluation  forms 

•  supervising  data  collection,  analysis,  and  interpretation 

•  preparing  diagnostic  reports  based  on  internal  feedback  data, 
with  recommendations  for  corrective  action 

•  monitoring  trends  in  the  results  of  various  internal  evaluation 
efforts  so  that  potential  problems  may  be  dealt  with  before  they 
become  critical. 

It  should  be  noted  that  the  above  functions  are  broad  in  scope  and 
will  require  accomplishing  a  great  many  specific  enabling  functions.  However, 
in  order  that  sufficient  flexibility  remain  for  meeting  unique  local  needs, 
the  details  of  accomplishing  broad  internal  evaluation  functions  should 
best  be  left  to  the  discretion  of  professional  CISO  personnel. 

Figure  3  depicts  possible  relationships  among  internal  evaluation, 
external  evaluation,  instructional  change,  and  graduate  performance.  The 
cycle  between  internal  evaluation  and  instructional  change  should  be  more 
or  less  continuous.  Shortcomings  in  the  instructional  process  uncovered  by 
an  ongoing  internal  evaluation  program  would  give  rise  to  corrective  changes 
in  the  instructional  process.  In  turn,  changes  in  the  instructional  process 
should  give  rise  to  changes  in  the  internal  evaluation  process,  in  order 
that  the  internal  evaluation  program  can  remain  tailor-made  for  that  local 
training  activity. 

While  the  cycle  of  changes  between  internal  evaluation  and  the  instruc¬ 
tional  process  should  repeat  frequently,  a  more  infrequently  occurring  cycle 
will  include  external  evaluation  and  changes  in  graduate  performance.  A 
formal  external  evaluation  can  provide  valuable  information  to  local  training 
activities  concerning  strengths  and  weaknesses  in  training.  However,  there 
are  limitations  to  the  usefulness  of  external  evaluation  results  as  they 
are  likely  to  be  delivered  to  training  activities.  First,  the  NAVEDTRACOM 
external  evaluation  system  is  designed  to  serve  all  training  activities. 

It  is  not  designed  to  serve  the  unique  needs  of  specific  training  activities 
(although  CNET  015  may  conduct  specific  evaluations  requested  by  a  local 
command).  Therefore,  some  translation  by  local  professional  personnel  of 
the  external  evaluation  results  is  both  desirable  and  necessary.  Second, 
although  external  evaluation  may  uncover  weaknesses,  it  will  not  typically 
disclose  solutions.  Thus,  the  external  evaluation  results  should  serve  to 
give  guidance  in  the  revision  of  internal  evaluation,  which  will  be  designed 
to  fit  the  unique  problems  of  the  local  activity  and  to  explore  possible 
solutions  to  existing  problems  as  well. 
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Figure  3.  Relationships  Among  Components  of  a  Quality  Assurance  Program 


The  link  between  external  evaluation  and  the  instructional  process  is 
graduate  performance.  Changes  in  the  instructional  process  should  result 
in  changes  in  graduate  performance;  changes  in  performance  should  result  in 
changes  in  external  evaluation  results.  Finally,  results  of  external  evalua¬ 
tion  should  lead  to  internal  evaluation  and,  when  necessary,  to  changes  in 
instructional  delivery. 

Instructional  Support  Functions.  Instructional  support  functions  recommended 
for  CISO  accomplishment  are  concerned  mainly  with  the  conduct  of  inservice 
training  for  local  personnel.  The  CISO  should  have  primary  responsibility 
for  conducting  inservice  in  various  professional  educational  areas,  including 
curriculum  design,  instructional  delivery  systems,  test  development,  and 
evaluation  techniques.  Such  inservice  programs  would  cross  departmental 
lines  and  would  be  relatively  independent  of  technical  subject  matter.  Also, 
CISO  personnel  could  offer  special  workshops  as  needed  in  areas  of  special 
interest  to  instructional  personnel.  Inservice  programs  that  are  designed 
for  a  specific  technical  subject  matter  are  probably  best  accomplished  within 
a  training  department.  However,  CISO  personnel  would  be  expected  to  serve 
in  a  consulting  role  similar  to  that  served  for  any  other  program  of  instruc¬ 
tion.  This  consultation  would  include  advice  concerning  curriculum  format. 
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instructional  delivery  and  evaluation  of  the  effectiveness  of  the  inservice 
program. 

Administrative  Functions.  As  noted  previously,  the  only  administrative 
type  functions  considered  appropriate  for  a  local  QA  office  (CISO)  are  those 
that  arise  as  a  natural  or  logical  consequence  from  the  performance  of  other 
more  technical  functions,  certain  functions  that  cut  across  departmental 
lines,  or  that  involve  outside  activities  from  a  quality  assurance  standpoint. 
Suggestions  concerning  delegating  administrative  functions  to  local  CISOs 
are  discussed  below. 

Liaison/Coordination.  The  CISO  could  properly  function  as  the  local  command's 
interface  with  external  agencies  on  matters  concerned  with  training  quality 
assurance  practices  and  programs  at  the  local  activity.  The  office  could 
serve  as  an  initial  point  of  contact  for  outside  activities  requesting  informa¬ 
tion  or  assistance  in  these  areas.  It  is  further  suggested  that  this  office 
serve  as  the  primary  point  of  contact  concerning  applying  educational  tech¬ 
nology  to  local  training.  As  part  of  this  function,  the  local  CISO  could 
provide  liaison  with  fleet  activities  and  other  training  units  for  these 
matters. 

Since  the  CISO  activities  and  interests  cross  departmental  lines,  there 
are  certain  other  coordinating  functions  that  might  logically  be  assigned 
to  these  offices.  These  include: 

•  coordinating  and  staging  IG  requests  involving  training  quality 
assurance  matters 

•  coordinating  and  sponsoring  visits  from  NAVEDTRACOM,  Navy,  and 
other  military  service  personnel  concerned  with  training  quality 
assurance 

•  reviewing  and  coordinating  requests  from  research  activities  for 
access  to  research  subjects,  technical  assistance,  or  student 
record  data  to  conduct  TECHTRACOM  sanctioned  studies. 

Compliance  with  Higher  Authority.  As  the  central,  local  repository  for  train¬ 
ing  technology  expertise,  the  CISO  should  review  all  local  training  instruc¬ 
tions  to  determine  their  currency  and  compliance  with  higher  authority  instruc¬ 
tions.  Appropriate  recommendations  should  be  made  to  the  local  commanding 
officer. 

As  part  of  a  continuing  quality  assurance  of  training  function,  the 
CISO  should  critically  review  all  practices  of  the  training  departments  and 
make  appropriate  recommendations  to  the  departments  for  maintaining  good 
quality  assurance  practices  and  procedures.  At  a  minimum,  the  CISO  should 
determine  that  training  departments  have  established  and  are  maintaining 
programs  required  by  higher  authority.  For  example,  the  CISO  should  ensure 
that  training  departments  conduct  annual  course  reviews,  collect,  record, 
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and  maintain  appropriate  records  on  students.  A  central  file  of  all  instruc¬ 
tions  could  be  maintained  by  this  office  and  CISO  personnel  should  assist  the 
training  departments  in  understanding  and  complying  with  the  instructions. 

Special  Administrative  Functions.  In  addition  to  the  functions  discussed 
above,  there  are  certain  other  special  administrative  functions  that  could 
be  performed  well  by  the  local  CISO.  It  is  recommended  that  the  CISO: 

1.  Coordinate  accreditation  requirements  and  review. 

Because  of  the  nature  of  personnel  training  and  the  interdepart¬ 
mental  role  it  serves,  the  CISO  seems  a  logical  choice  for  performing  all 
necessary  coordination/review  for  accreditation  of  local  courses. 

2.  Recommend  candidates  for  Master  Training  Specialist  designation. 

A  logical  outgrowth  of  functions  performed  to  evaluate  instructors 
is  the  recommendation  of  candidates  for  the  Master  Training  Specialist  designa¬ 
tion.  This  practice  is  not  currently  followed;  however,  CNTECHTRA  staff 
submit  that  it  may  be  adopted  by  the  command.  Related  to  this  function  is 
the  designation  of  "Instructor  of  the  Quarter."  Many  CISOs  now  assist  in 
this  activity  and  it  is  recommended  as  a  continuing  CISO  function. 

3.  Advise  the  commanding  officer  on  CISO  civilian  affairs. 

The  senior  civilian  education  or  training  specialist  assigned  to 
the  CISO  should  routinely  assist  and  advise  the  local  commanding  officer  on 
matters  affecting  the  professional  civilian  work  staff.  This  should  include 
hiring,  promotions,  demotions,  transfers,  discharges,  and  training,  as  they 
pertain  to  civilians  attached  to  the  CISO. 

Special  Training  Support  Functions.  There  are  other  functions  that  should 
be  the  province  of  the  CISO  but  that  do  not  fall  neatly  into  any  other  pre¬ 
viously  mentioned  category.  These  functions  should,  for  the  most  part,  be 
the  concern  of  civilian  education  specialists  within  the  local  training 
command.  The  senior  education  specialist  should  provide  continuity  to  the 
local  command  on  all  matters  of  curriculum,  instructional  programs,  and 
evaluation  programs.  Education  specialists  should  also  monitor  current 
developments  in  training  methodology,  technology,  and  research,  with  a  view 
toward  possible  useful  applications  at  the  local  command.  Education  special¬ 
ists  may,  as  situations  and  needs  arise,  conduct  or  supervise  the  conduct 
of  special  research  projects  or  studies  in  training  areas  of  interest. 

They,  or  other  qualified  CISO  staff,  could  also  attend  and  monitor  pilot 
courses  to  provide  constructive  criticism  and  assistance  in  improving  or 
finalizing  the  courses. 

ORGANIZATION  AND  STAFFING  CONSIDERATIONS.  Suggestions  and  comments  relevant 
to  CISO  organization  and  staffing  are  given  below. 

Organization.  Since  the  routine  accomplishment  of  many  of  the  functions 
suggested  involves  various  levels  of  a  local  command,  it  is  recommended 
that  the  CISO  report  directly  to  the  executive  officer.  The  CISO  may  be 
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headed  by  an  officer  or  by  the  senior  education  specialist.  However,  unless 
the  officer  has  an  adequate  background  in  educational  technology  and/or 
evaluation,  the  senior  education  specialist,  because  of  his  specialized 
knowledge  and  skills,  should  have  cognizance  over  the  majority  of  CISO  func¬ 
tions.  If  an  officer  does  head  the  CISO,  he/she  should  be  at  least  equal 
in  rank  to  the  training  department  heads.  A  desirable  requirement  for  naval 
officers  assigned  as  CIS  officers  is  to  be  designated,  or  working  towards 
designation,  as  an  education  and  training  management  subspecialist.  Regard¬ 
less  of  who  heads  the  CISO,  it  should  be  recognized  that  the  primary  mission 
of  the  CISO  should  be  applying  educational  technology  and  evaluation  techniques 
to  local  training  problems.  The  training  activity  CISO  should  be  the  central 
repository  of  educational  technology  and  evaluation  expertise.  Accordingly, 
it  is  recommended  that  all  education  and  training  specialists  be  assigned 
to  this  office.  When  needed,  they  could  provide  assistance  to  individual 
training  departments.  CISO  personnel  should  be  allowed  direct  liaison  with 
instructional  personnel,  since  there  should  be  a  great  deal  of  cooperative 
work.  Since  the  two  broad  missions  of  the  CISO  are  curriculum  support  and 
evaluation,  and  since  these  areas  represent  different  skills  and  techniques, 
there  should  be  two  divisions  to  the  CISO--curriculum  support  and  evaluation. 

Figure  4  depicts  proposed  CISO  relationships.  The  relationships  indicated 
by  solid  lines  in  figure  4  represent  tasking  authority.  Thus,  the  CISO 
branches  report  to  and  are  tasked  by  the  CIS  officer,  who,  in  turn,  reports 
to  and  is  tasked  by  the  commanding/executive  officer.  The  relationships 
indicated  by  dotted  lines  represent  direct  liaison  between  the  CISO  and  the 
local  training  departments  and  between  the  CISO  and  CNTECHTRA  N63  and  other 
interested  activities.  The  proposed  relationship  with  CNTECHTRA  N63  is 
discussed  further  below.  Although  figure  4  focuses  on  CNTECHTRA,  it  is 
suggested  that  CISOs  under  other  CNET  Functional  Commanders  be  organized  in 
similar  ways. 

Staffing.  The  basic  functions  suggested  for  performance  by  local  CISOs 
represent  a  minimum  set  which  should  be  performed  in  the  interest  of  estab¬ 
lishing  and  maintaining  high  quality  training  programs.  They  should  be 
performed  at  all  training  activities  regardless  of  the  size  ( i . e . ,  numbers 
of  courses,  student  throughput)  of  the  activity.  With  the  exception  of 
certain  administrative  functions,  it  is  recommended  that  they  be  performed 
by  individuals  trained  in  instructional  design  and  in  evaluation  techniques. 
Ideally,  the  CISO  staff  should  also  contain  individuals  with  instructional 
skills  who  can  provide  training  for  other  local  personnel. 

In  general,  there  are  three  factors  to  consider  when  staffing  a  CISO. 
First,  there  must  be  personnel  knowledgeable  in  applying  educational  tech¬ 
nology  to  training  problems.  These  skills  would  most  likely  be  present  in 
an  education  or  training  specialist.  However,  it  is  the  possession  of  knowl¬ 
edge  and  skills  that  is  important,  not  the  possession  of  degrees  or  titles. 
Second,  there  must  be  personnel  knowledgeable  in  the  techniques  of  evaluating 
training  programs,  such  as  someone  with  graduate  training  in  social  science 
methods.  Again,  however,  the  crucial  issue  is  possession  of  skills  and 
knowledge,  not  degrees.  Third,  the  CISO  must  have  access  to  SMEs  for  the 
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Figure  4.  Proposed  CISO  Relationship  within  the  Local 
Command  and  with  other  Activities 
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ratings  represented  by  the  courses  taught  at  the  local  riool.  It  is  prob¬ 
ably  not  necessary  that  these  SMEs  belong  solely  to  CISO;  they  could  be 
shared  with  instructional  departments  or,  perhaps,  rotated  back  and  forth. 

Normally,  requisite  capabilities  for  training  quality  assurance  are 
associated  with  education  specialists  (1710  series)  and  training  specialists 
(1712  series)  and,  occasionally,  education  technicians  (1702  series).  How¬ 
ever,  there  is  no  reason  why  necessary  skills  cannot  be  obtained  by  assigned 
military  personnel  through  appropriate  combinations  of  experience  and  train¬ 
ing.  It  is  recommended  that  selected  military  staff  be  designated  by  their 
commands  to  receive  such  training  and  be  assigned  CISO  duties  independently 
of  instructional  duties.  In  this  context,  it  is  further  recommended  that 
consideration  be  given  to  the  creation  of  an  educational  and  training  career 
field  for  enlisted  personnel. 

Undoubtedly,  local  commands  would  prefer  that  the  capabilities  required 
to  perform  all  necessary  QA  functions  be  resident  within  the  given  command, 
and  it  is  recommended  that  all  training  activities  be  assigned  a:  least  one 
1710  or  1712  for  CISO  work.  However,  the  number(s)  of  such  individuals  who 
are  actually  needed  at  local  levels  cannot  be  easily  determined.  Optimum 
staffing  for  any  one  command  is  a  complex  problem  for  which  there  is  no 
simple  solution.  Staffing  requirements  are  a  function  of  QA  workload.  This 
workload  is  affected  by  the  number  of  courses  and,  in  the  case  of  curriculum 
design/redesign  functions,  by  the  frequency  and  extent  of  change  required 
for  courses.  Staffing  requirements  are  also  affected  by  the  skills  and 
capabilities  of  individuals  already  assigned  to  a  command  and  by  the  way  in 
which  assigned  talent  is  used. 

Given  the  currently  austere  funding  situation  within  the  NAVEDTRACOM, 
additional  billets/positions  for  necessary  specialized  talent  may  be  extremely 
difficult  for  local  commands  to  acquire  (although  they  certainly  should 
attempt  to  do  so  when  this  is  warranted).  Consequently,  local  commands,  at 
least  in  the  short  run,  should  be  encouraged  to  obtain  necessary  training 
through  CNTECHTRA  for  assigned  military  and  civilian  staffs.  Travel  funds 
should  be  available  for  this  purpose.  A  second  alternative  is  to  request 
direct  assistance  from  CNTECHTRA  for  accomplishing  necessary  functions  (see 
also  pages  76  and  77). 

It  is  recommended  that,  at  the  very  minimum,  local  CISO  staff  (military 
or  civilian)  be  given  appropriate,  necessary  experience  or  training  and 
that  the  local  CISO  perform  the  evaluation  functions  listed  previously  with¬ 
out  direct  outside  assistance  (other  than  training).  It  is  also  recommended 
that  appropriate  administrative  functions  listed  previously  be  performed  by 
the  local  CISO. 

Requirements  for  outside  assistance  most  often  will  arise  for  perform¬ 
ance  of  curriculum  design/redesign  functions  and  certain  instructional  support 
functions  as  influenced  by  local  CISO  staff  capabilities.  Outside  support 
may  take  forms  ranging  from  simply  providing  training  to  assigned  local 
staff  through  the  accomplishment  of  significant  portions  of  a  design  effort 
by  an  outside  activity  (e.g.,  perform  all  the  necessary  task  analysis  and 
provide  results  to  the  school). 
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It  Is  not  possible  to  discuss  all  of  the  possible  contingencies,  shared 
responsibilities,  or  nature  of  interactions  that  could  occur  for  curriculum 
design/redesign  efforts.  It  is  suggested  that  specific  arrangements  to 
accomplish  necessary  work  be  made  on  a  case  by  case  basis.  In  this  regard, 
it  is  further  suggested  that  the  local  command  prepare  a  course  development 
plan  that  identifies: 

•  the  course(s)  requiring  development  work 

•  the  nature  and  extent  of  work  required 

«  local  plans  for  performing  the  work  to  include: 

..  identification  of  individuals  who  will  be  assigned  to  do  the 
work 

..  skills  and  qualifications  of  these  individuals 
..  amount  of  time  each  will  devote  to  given  segments 
..  duration  of  planned  effort  and  impact 
..  specific  needs  for  assistance  in  the  effort(s). 

These  development  plans  could  be  forwarded  to  CNTECHTRA  via  the  TPC  or 
directly  to  N-63.  Dialogues  between  appropriate  CNTECHTRA  and  school  per¬ 
sonnel  could  then  focus  on  clarifying  specific  assistance  needs.  Using  the 
plan  as  a  basis,  CNTECHTRA  N-63  could  then  arrange  for  obtaining  required 
assistance  (e.g.,  from  the  TPC,  N-63  staff,  IPDCs,  contract).  It  is  recom¬ 
mended  that  in  most  cases  ultimate  responsibility  for  the  course  development 
effort  remain  at  the  local  level  where  the  subject  matter  expertise  resides. 
Thus,  the  outside  activities,  in  most  instances,  will  be  serving  a  supporting 
role.  In  addition  to  the  planning  purpose  of  the  suggested  development 
plan,  this  document  could  also  be  used  in  conjunction  with  the  "account¬ 
ability  system"  discussed  below. 

COMMAND  INTERFACES.  The  CISO  should  remain  an  integral  part  of  the  activity 
that  it  serves;  it  should  not  become  an  agent  of  higher  authority.  However, 
it  is  recommended  that  stronger  ties  be  formally  established  between  CNTECHTRA 
N-63  and  local  CISOs.  CNTECHTRA  N-63  has  overall  responsibility  within  the 
TECHTRACOM  for  many  of  the  same  functions  that  CISOs  perform  locally.  Direct 
tasking  authority  from  CNTECHTRA  N-63  to  the  local  CISO  is  not  advocated; 
this  would  in  fact  make  the  CISO  an  agent  of  higher  authority  and  would 
probably  have  undesirable,  disruptive  effects.  A  dotted  line  (ADDU)  relation¬ 
ship  between  N-63  and  the  local  CISO,  similar  to  that  of  local  schools  to 
CNTECHTRA  TPCs,  is  envisioned.  CNTECHTRA  N-63  should  serve  as  an  open, 
direct  link  between  CISOs  and  the  TECHTRACOM  for  guidance  or  assistance  in 
training  quality  assurance  matters.  Specific  recommendations  are  that  N-63: 

•  assist  CISOs  in  identifying  and  obtaining  appropriate  personnel 

or  other  resources  needed  for  effectively  accomplishng  local  quality 
assurance  functions.  This  assistance  should  concern  both  permanent 
and  temporary  resource  augmentation. 
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•  provide  or  obtain  inservice  training  for  local  CISO  staff.  This 
training  could  be  conducted  at  various  locations  by  N-63  staff  or 
by  other  agencies  arranged  for  by  N-63.  Appropriate  funding  will 
be  required. 

•  advise  CISOs  of  planned  higher  authority  initiatives  that  may 
affect  local  training  quality  assurance  practices  and  assist  in 
preparing  for  new  requirements. 

•  provide  direct  technical  assistance  to  the  CISOs  for  specialized 
problems.  In  addition  to  providing  training  for  CISO  staff,  N-63 
staff  could  assume  direct  responsibility  for  executing  portions 
of  the  quality  assurance  work  required  at  the  local  level. 

•  monitor  actively,  through  the  CISOs,  all  training  quality  assurance 
operations  at  local  levels. 

Implementing  the  above  recommendations  will  require  augmenting  the 
N-63  staff  and  appropriate  funding  for  performing  the  assistance  functions 
described.  CNET  may  wish  to  consider  if  CNTECHTRA  N63  should  provide  similar 
assistance  to  CISOs  reporting  to  other  Functional  Commands. 

ENFORCEMENT/ACCOUNTABILITY.  The  CISO  concept  as  a  means  of  assuring  the 
quality  of  local  schoolhouse  training  is  affected  by  a  number  of  factors. 

These  factors  limit  CISO  operations  and  affect  the  degree  of  effectiveness 
for  training  quality  assurance  that  these  offices  can  achieve.  It  was  stated 
in  section  V  that  some  method(s)  of  assuring  that  local  commands  properly 
employ  CISO  personnel  and  otherwise  strive  to  achieve  a  high  level  of  training 
quality  is  desirable.  In  this  regard,  it  is  recommended  that  an  accountability 
system  be  developed  and  implemented.  This  system  could  provide  information 
concerning  local  quality  assurance  practices  and  problems  to  the  activity 
commanding  officer.  This  information  should  also  be  available  to  higher 
authority  for  review  of  local  actions.  To  avoid  undue  infringement  on  the 
local  commander's  authority  and  prerogatives,  it  is  suggested  that,  for  the 
most  part,  internal  mechanisms  visible  to  higher  authority  be  considered 
for  accountability  purposes.  An  accounting  procedure  for  local  commanding 
officer  use  is  one  possibility;  for  example,  a  checklist  containing  functions 
assigned  to  CISOs  could  be  used  to  record  requests  or  needs  for  educational 
technology  assistance,  reasons  for  not  receiving  it,  CISO  initialing  to 
indicate  participation  in  efforts.  These  checklists  should  then  be  available 
for  inspection  by  higher  authority  during  audits,  reviews,  etc.,  or  when 
external  feedback  indicates  that  significant  numbers  of  course  graduates 
are  considered  inadequate.  Consideration  could  also  be  given  to  a  higher 
authority  instruction  requiring  the  schools  to  develop  and  submit  plans 
concerning  how  educational  technology  expertise  will  be  applied  at  the 
schools.  Periodic  audits  for  compliance  or  for  providing  directed  assistance 
could  be  accomplished  by  CNTECHTRA  N-63.  In  addition,  N-63  could  also  be 
assigned  responsibility  for  reviewing  the  audit  trail  for  ISD  programs  con¬ 
ducted  by  local  activities.  It  is  recognized  that  an  additional  means  of 
obtaining  accountability  is  inherent  in  the  CNET  Training  Appraisal  System 
(TAS)  results.  Interpreting  the  results  of  those  feedback  findings  in  rela¬ 
tion  to  operations  at  the  local  schools  is  complex,  however. 
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SECTION  VIII 
RECOMMENDATIONS 

Work  performed  during  the  study  led  to  the  conclusion  that  the  CISO 
concept  is  basically  sound.  A  number  of  positive  features  and  benefits  for 
training  are  associated  with  the  notion  of  an  office  within  a  local  training 
activity  that  is  specifically  charged  with  responsibility  for  assuring  train¬ 
ing  quality.  However,  the  CISO  concept  has  not  been  well  implemented  within 
the  Command.  CISOs  across  the  NAVEDTRACOM  are  highly  variable  organization¬ 
ally  and  in  terms  of  the  functions  they  perform.  Further,  a  number  of  local 
conditions  limit  the  potential  effectiveness  of  these  offices.  These  include 
lack  of  manpower  (both  numbers  and  types  of  skills  required)  and  other  factors 
which  affect  local  utilization  of  assigned  CISO  staff. 

Two  sets  of  recommendations  are  offered  for  improving  CISO  contributions 
to  training  quality.  The  first  set  concerns  the  total  CISO  concept.  These 
recommendations,  presented  in  section  VII  of  this  report,  are  provided  in 
the  form  of  a  proposed  charter.  Specific  elements  of  the  proposed  charter 
concern  the  following  areas: 

•  functions  that  are  appropriate  for  CISO  staff  to  perform 

•  organization  and  staffing  of  CISOs 

•  interfaces  between  locil  CISOs  and  CNTECHTRA 

•  mechanisms  for  ensuring  the  performance  of  training  quality  assurance 
functions  at  local  command  levels. 

The  second  set  of  recommendations  consists  of  separate  actions  that 
could  be  taken  to  assist  generally  the  cause  of  enhancing  the  quality  of 
local  training.  These  are  listed  below: 

1.  Develop  a  short  course  covering  purposes,  principles,  and  procedures 
of  ISD  and  require  that  at  least  all  officers  reporting  to  training  command 
billets  complete  the  course. 

2.  Develop  a  handbook  based  on  NAVEDTRA  110  for  use  of  CISO  personnel 
to  provide  guidance/instructions  concerning  tasks  assigned  to  them.  This 
handbook  should  contain  information  such  as  how  to  conduct  task  analyses, 
develop  tests,  requirements  and  procedures  for  instructor  evaluation.  In 
addition  to  providing  guidance,  the  handbook  should  also  identify  sources 
where  more  detailed  information  on  specific  topics  can  be  found. 

3.  Develop  standardized  courses  in  training  quality  assurance  areas. 
These  courses  could  be  used  for  training  of  CISO  personnel.  They  could 
also  be  used  by  CISOs  for  inservice  training  of  other  locally  assigned 
personnel . 


4.  Rewrite  instructions  governing  CISOs.  Currently,  several  CNET 
and  CNTECHTRA  instructions  assign  tasks  to  CISOs.  It  is  recommended  that 
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these  taskings  be  appropriately  consolidated  and  that  one  instruction,  with 
cross  references  as  necessary,  be  wri-ten  and  promulgated  for  CISO  guidance. 

5.  Establish  a  direct  interface  between  CNTECHTRA  and  local  CISOs 
for  defined  purposes.  It  is  specifically  recommended  that  CNTECHTRA  N63  be 
appropriately  tasked,  staffed,  and  funded  to: 

•  maintain  direct  cognizance  over  local  CISO  operations 

•  assist  local  CISOs  in  resolving  problems  and  obtaining  resources 
or  training  needed  to  perform  assigned  work. 

6.  Establish  proficiency  billets  for  military  personnel  to  be  assigned 
to  CISOs. 
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APPENDIX  A 

CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE  FUNCTIONS 


This  appendix  lists  functions  which  CISOs  are  expected  to  perform. 

The  functions  were  compiled  from  current  higher  authority  instructions  and 
from  local  CISO  and  CNTECHTRA  staff  suggestions. 
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CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE  FUNCTIONS 

Functions  which  CISOs  are  expected  to  accomplish  are  listed  below. 

The  list  is  divided  into  three  parts.  Functions  1  through  97  are  those 
assigned  CISOs  by  CNET  Instruction  1540.6  and  CNTECHTRA  Instruction  1540.40. 
Functions  98  through  106  were  suggested  for  CISO  accomplishment  by  local 
staff  at  the  Naval  Submarine  School.  Functions  107  through  125  were  suggested 
by  CNTECHTRA  staff. 

Functions  whose  numbers  are  preceded  by  an  "X"  are  those  which  for 
various  reasons  the  TAEG  project  staff  considered  inappropriate  for  CISO 
accomplishment  in  the  future.  In  some  instances  they  represent 
subfunctions  which  need  not  be  separately  stated.  Functions  retained  for 
consideration  as  future  CISO  tasks  are  preceded  by  other  letter  codes.  The 
functions  retained,  were  grouped  according  to  the  following  letter  codes: 

A  -  Curriculum  Design/Redesign 

B  -  Instructional  Support 

C  -  Evaluation 

D  -  Administrative 

E  -  Special  Training  Support 

In  most  instances  where  functions  were  considered  inappropriate  for 
CISOs,  the  activity  or  activities  recommended  for  the  function  is 
identified  in  parentheses  as  follows: 

(TD)  -  Training  Department 

(TPC)  -  Training  Program  Coordinator 

(Admin)  -  Administrative  Section 

(CNTECHTRA)  -  Chief  of  Naval  Technical  Training 

(CO)  -  Training  Activity  Commanding  Officer 
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CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE  FUNCTIONS 

CNETINST  1540.6  and  CNTECHTRAINST  1540.40 

XI.  Monitor  procurement  of  training  devices  (TPC) 

X2.  Monitor  assignment  of  training  devices  (TPC/TD) 

X3.  Monitor  maintenance  of  training  devices  (TD) 

A4.  Review  and  approve  all  locally  produced  training  publications 

X5.  Develop  plans  for  activating  emergency  Navy  curricula  (TPC/TD) 

X6.  Work  with  CMI  systems  manager  for  CMI  course  development  and 
implementation  (TPC) 

X7.  Monitor  development  and  maintenance  of  CMI  course  files  (TPC/TD) 

D8.  Provide  liaison  with  fleet  activities  and  other  training  units 

E9.  Provide  continuity  for  the  command  on  all  curriculum  and 
instructional  programs  and  evaluation  efforts 

E10.  Monitor  current  developments  in  training  methodology,  technology, 
and  research 

XII.  Recommend  improvements  in  plans,  policies,  or  techniques  (not 
uniquely  CISO) 

B12.  Conduct  inservice  in  curriculum  design,  instructional  delivery 
systems,  and  internal  evaluation 

A13.  Task  analysis 

A14.  Review  curricula 

A15.  Revise  curricula 

A16.  Approve  curricula 

A17.  Design  course  materials 

A18.  Monitor  writing  of  learning  objectives 

A19.  Select  instructional  delivery  systems 

A20.  Review  curriculum  control  documents 
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1 


Acquire  course  documentation 
Maintain  course  documentation 

«  Part  of  Normal  Review 

Acquire  course  approval  data 

Maintain  course  approval  data 

Revise  courses  based  on  evaluation 

Monitor  administration  of  Navy  Training  Plans  (TPC) 

Monitor  implementation  of  Navy  Training  Plans  (TPC) 

Conduct  research  projects  and  studies  in  training  areas  (CNTECHTRA) 

Develop  curricula 

Develop  training  materials 

Review  courses 

Assure  adherence  to  applicable  instructions  (CO) 

Forward  revised  curricula  to  functional  commanders  for  approval  (CO) 
Visit  classrooms  (Step  in  doing  other  functions) 

Evaluate  instruction 

Debrief  newly  assigned  instructors 

Develop  instructor  inservice  training 

Supervise  instructor  inservice  training  (TD  with  CISO  in  support  role) 
Maintain  central  technical  library  (TD) 

Maintain  course  curriculum  model  (TD) 

Maintain  bank  of  test  items  (TD) 

Prepare  examinations  (TD) 

Analyze  test  data 
Improve  test  items 

Prepare  internal  feedback  instruments 
Prepare  external  feedback  instruments 
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C47.  Supervise  feedback  data  collection 

C48.  Supervise  feedback  data  analysis 

C49.  Supervise  feedback  interpretation  and  reporting 

X50.  Prepare  Training  Appraisal  Plans  (TAPs)  (No  longer  required) 

X51.  Monitor  implementation  of  approved  TAPs  (No  longer  required) 

X52.  Update  TAPs  (No  longer  required) 

C53.  Prepare  student  critique  program 

C54.  Administer  student  critique  program 

C55.  Summarize  evaluation  data 

C56.  Disseminate  evaluation  data 

C57.  Monitor  attrition  data 

C58.  Study  causes  of  attrition 

C59.  Develop  plans  to  reduce  attrition 

C60.  Monitor  setback  data 

C61 .  Study  causes  of  setbacks 

C62.  Develop  plans  to  reduce  setbacks 

C63.  Standardize  student  critiques 

C64.  Standardize  course  reviews 

X65.  Standardize  classroom  visits  (Requirement  not  clear) 

X66.  Coordinate  course  reviews  (TPC/TD  -  Scheduling(?)) 

X67.  Coordinate  classroom  visits  (TD) 

C68.  Evaluate  instructors 

X69.  Evaluate  learning  supervisors  (Part  of  68) 

C70.  Evaluate  course  review  procedures 

X71.  Standardize  graduate  questionnaires  (No  longer  a  school  requirement) 
X72.  Standardize  supervisor  questionnaires  (No  longer  a  school  requirement) 
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C73.  Standardize  information  from  follow-on  schools 

C74.  Standardize  information  from  instructors  returning  from  the 
fleet 

C75.  Standardize  information  from  students  returning  from 
fleet 

C76.  Standardize  information  from  other  activities 

C77.  Coordinate  administration  of  internal  feedback  data  collection 

X78.  Coordinate  administration  of  supervisor  questionnaires  (No  longer 
done) 

X79.  Coordinate  administration  of  information  from  follow-on  schools  (Not 
clear) 

X80.  Coordinate  administration  of  information  from  instructors  returning 
from  fleet  (Not  clear) 

X81.  Coordinate  administration  of  information  from  students  returning 
from  fleet  (Not  clear) 

X82.  Coordinate  administration  of  information  from  other  activities  (Not 
clear) 

C83.  Forward  outstanding  methods  for  obtaining  feedback  to  CNTECHTRA 

X84.  Standardize  data  from  training  departments  on  student/instructor 
ratios  (TD/Admin) 

X85.  Standardize  data  from  training  departments  on  instructor 
utilization  (TD/Admin) 

X86.  Standardize  data  from  training  departments  on  class  size  (TD/Admin) 

X87.  Standardize  data  from  training  departments  on  convening  frequency 
(TD/Admin) 

X88.  Analyze  master  schedules  (TD/Admin) 

X89.  Analyze  master  schedule  data  analysis  worksheets  (TD/Admin) 

X90.  Analyze  student  instructor  ratios  (TD/Admin) 

X91.  Analyze  instructor  utilization  reports  (TD/Admin) 

892.  Train  personnel  in  training  departments  and  courses 

C93.  Evaluate  training  departments'  inservice  programs  (If  training  Is  not 
done  by  Cl SO) 


Subsumed 

under 

other 

functions 
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X94.  Schedule  annual  course  reviews  (TPC/TD) 

X95.  Provide  course  review  schedules  to  CNTECHTRA  (TPC/TD) 

C96.  Acquire  data  on  instructor  qualifications  (As  necessary) 

C97.  Maintain  data  on  instructor  qualifications  (As  necessary) 

U.S.  Naval  Submarine  School 
E98.  Attend/Monitor  all  pilot  courses 

X99.  Conduct  special  projects  (e.g..  Manpower  Survey,  Accounting  Feedback 
Reports) 

B100.  Establish/Monitor  Remediation  Courses 

C101.  Direct  Master  Training  Specialist  Program 

C102.  Direct  Instructor  of  the  Quarter  Program 

B103.  Direct/guide  specialized  training/responsive  training 

C104.  Review  all  student  critiques 

X105.  Head  civilian  review  board  (CO) 

X106.  Coordinate  instructor  utilization  program  and  make  reports  (TD) 

CNTECHTRA  Staff  Suggestions 

C107.  Review  instructional  development  process/instructional 

materials/supporting  audit  trail  materials  from:  IPDCs,  Contract 
Training  Curricula,  and  Instructors 

X108.  Monitor  ACRs  and  ensure  action  is  taken  for  inadequacies  (Included  in 
other  areas) 

A109.  Advise/assist  curricula  developers  with  the  integration  of  special 
areas  such  as  PQS,  3M,  Safety 

C110.  Sample  for  trends  established  by  end  of  course  student  critiques 

Clll.  Conduct  instructional  programs  in  the  areas  of  review  and  testing 
procedures  and  test  results 

D112.  Review  local  training  instructions  for  currency  and  compliance  with 
higher  authority  instructions 

X 1 13 .  Distribute  higher  authority  instructions  (Admin) 
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B114.  Ensure  and  assist  inservice  training 

A11E.  Coordinate  interservice  training  requirements  relative  to  curricula 

B116.  Offer  special  assistance  for  workshops  on  curricula  development  which 
cross  school  lines,  require  special  SMEs,  have  specified  time  limits 

D117.  Coordinate  accreditation  requirements  and  review 

XI 18.  Act  as  administration  unit  for  the  training  instruction  department 
(recommended  for  deletion  by  CNTECHTRA  staff) 

XI 1 9 -  Act  as  advisor  for  Course  Curriculum  Model  Manager  (CCMM)  in  control 
or  management  (TO/TPC) 

X120.  Coordinate  review  of  Navy  Train  ng  Plans  (TPC) 

X 121 .  Evaluate/analyze  feedback  (covered  elsewhere) 

C122.  (Desirable— not  mandatory)  Advise  and  assist  the  schoolhouse  on 
external  appraisal  programs  for 

a.  Level  I 

b.  Level  II 

c.  Level  III 

d.  Instructor  returning  f *om  fleet 

D123.  Act  as  sponsor  for  CNTECHTRA  representative 

C 124 .  Coordinate  and  stage  relevant  Ii  requests,  research  subject  requests, 
special  studies,  seminars,  etc. 

C 125 .  Recommend  candidates  for  Master  Training  Specialist  Designation 
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APPENDIX  B 

SAMPLE  QUESTIONNAIRE  PACKAGE 

This  appendix  contains  a  copy  of  the  cover  letter  and  two  of  the  four 
questionnaires  used  to  obtain  information  for  this  study.  The  short  form 
questionnaire  for  commanding  officers  and  executive  officers  is  virtually 
identical  to  the  short  form  questionnaire  sent  to  CIS  officers  and  training 
department  heads.  Therefore,  a  copy  of  these  two  questionnaires  is  not 
included  in  this  appendix.  Minor  changes  in  wording  were  made  to  reflect 
the  different  positions  of  the  recipients. 
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From:  Director,  Training  Analysis  and  Evaluation  Group 
To:  Commanding  Officer 

Subj:  Curriculum  and  Instructional  Standards  Office  Study 
Ref:  (a)  CNET  ltr  N-53  of  25  June  1980  (NOTAL) 

Enel:  (1)  CIS  Office  Survey  for  Commanding  Officers  or  Executive  Officers 

*(2)  CIS  Office  Survey  for  Training  Department  Heads/Regimental 

Commanders 

*(3)  CIS  Office  Survey  for  CIS  Officer 

(4)  CIS  Office  Survey  for  All  CISO  Personnel 

1.  At  the  request  of  the  Chief  of  Naval  Technical  Training,  the  Training 
Analysis  and  Evaluation  Group  (TAEG)  has  been  tasked  (reference  (a))  to 
conduct  a  short  term  study  of  the  Curriculum  and  Instructional  Standards 
Office  (CISO)  concept.  Your  assistance  is  requested  for  this  effort. 

2.  The  CNET  tasking  requires  that  the  TAEG  assess  current  CISO  capabili¬ 
ties  for  performing  various  training  development,  support,  and  evaluation 
functions.  For  this,  both  factual  and  opinion  data  are  needed  from  local 
school  personnel  who  either  perform  CIS  Office  duties  or  who  are  directly 
affected  by  CISO  operations/requirements. 

3.  Enclosures  (1)  through  (4)  contain  survey  forms  designed  for  obtaining 
required  information.  It  is  requested  that  these  forms  be  distributed  to 
individuals  occupying  billets/positions  identified  on  each  cover  sheet.  To 
preserve  anonymity  of  respondents  and  to  promote  candid  responses  and 
comments,  it  is  requested  that  individuals  complete  the  forms  independently 
and  return  them  directly  to  the  TAEG  in  the  envelopes  provided.  Completed 
forms  should  be  returned  within  10  working  days  after  receipt. 


*Not  included  in  this  appendix. 
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CIS  OFFICE  SURVEY 
for 


Commanding  Officers 
or 

Executive  Officers 


This  form  should  be  completed  by  either  the  CO 
or  the  XO  of  your  activity.  Please  indicate 
below  who  completed  the  form. 


Commanding  Officer 


Date 


Executive  Officer 


Enclosure  (1) 
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11.  Ensure  appropriate  use  of 
evaluation  results 


III.  Please  answer  the  following  questions  by  checking  the  most  appropriate  response. 
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Kossuth. 


Please  use  the  space  below  to  make  any  additional  comments  or  suggestions. 
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CIS  OFFICE  SURVEY 
for 

All  CISO  Personnel 


The  Training  Analysis  and  Evaluation  Group  (TAEG),  at 
the  request  of  CNET  and  CNTECHTRA,  is  conducting  a 
study  of  CIS  Offices.  This  survey  is  part  of  that  study. 

This  survey  is  to  be  filled  out  by  all  personnel  assigned 
to  CISO  duties  by  the  local  command.  Individual  responses 
will  not  be  disclosed  to  anyone  outside  the  TAEG.  All 
data  will  be  summarized  in  appropriate  tables  or  charts. 


Date 


Enclosure  (4) 
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CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE  SURVEY 

Instructions 

This  survey  form  is  intended  for  all  Curriculum  and  Instructional 
Standards  Office  personnel  occupying  designated  CIS  Office  billets  or 
positions.  The  form  is  divided  into  five  sections.  Complete  the  form 
independently  of  other  personnel.  Brief  instructions  for  each  section  are 
given  below. 

Section  I.  This  section  asks  for  information  concerning  your  educational 
and  work  background.  Answer  each  question  to  the  best  of  your  ability. 

Section  II.  This  section  concerns  resources  that  CIS  Offices  may  need 
to  carry  out  their  functions.  You  are  asked  to  make  judgments  about  priorities 
and  usefulness.  Please  give  your  best  estimates. 

Section  III.  This  section  asks  you  about  your  contacts  and  communications 
with  other  CISO  personnel  and  with  personnel  outside  the  CIS  Office.  When 
asked  for  average  frequency  of  contact,  please  choose  the  answer  that  is  most 
accurate. 

Sections  IV  and  V.  These  two  sections  deal  with  functions  that  CIS  Offices 
may  perform.  The  functions  listed  are  identical  in  both  sections,  but  the 
questions  across  the  top  are  different  in  each  section.  Please  read  the 
questions  across  the  top  and  answer  each  question  for  each  function.  Answer 
the  questions  from  the  point  of  view  of  what  you  personally  do  as  a  member  of 
the  CIS  Office. 

After  you  have  completed  the  form,  please  place  it  in  the  addressed 
envelope  and  return  it  directly  to: 

Director 

Training  Analysis  and  Evaluation  Group 
Naval  Training  Center 
Orlando,  FL  32813 

Attn:  CISO  study 
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CURRICULUM  AND  INSTRUCTIONAL  STANDARDS  OFFICE  SURVEY 


SECTION  I.  Respondent  Data 

a.  Series  (civilian)  or  NEC/NOBC  -  _  _ 

b.  Grade  (civilian)  or  rate/rank  -  _ 

c.  Current  position  or  billet  title  -  _ 

d.  Number  of  years/months  in  current  position  -  _  /  _ 

e.  Anticipated  length  of  stay  In  current  position  in  years/months  -  _  / 

Educational  Profile 

f.  What  is  the  highest  educational  level  you  have  attained?  (check  one) 

(1)  did  not  graduate  from  high  school 

(2)  high  school  diploma  or  G.E.D. 

(3)  some  college  or  technical  training  beyond  high  school,  but  not 
bachelor's  degree 

(4)  graduated  from  college  (B.A.,  B.S.,  or  other  bachelor's  degree) 

(5)  some  graduate  school  but  no  graduate  degree 

(6)  Master's  degree 

(7)  Ed.S.,  Ed.D. ,  Ph.D.,  or  other  post  master's  or  professional  degree 


g.  If  you  received  a  bachelor's  degree,  put  a  1  by  the  response  that  indicates 
your  major  and  a  2  by  the  response  that  indicates  your  minor. 

_  (1)  Education 

_  (2)  Social  science  (psychology,  sociology,  etc.) 

_  (3)  Technical  or  scientific  (math,  physics,  engineering,  etc.) 

_  (4)  Business  administration 

_  (5)  Other  (specify)  _ 

_  (6)  Other  (specify)  _ 


h.  If  you  received  a  graduate  degree,  put  a  1  by  the  response  that  Indicates 
your  major  and  a  2  by  your  minor. 

_  (1)  Education 

_  (2)  Social  Science 

_  (3)  Technical  or  scientific 

___  (4)  Business  administration 

_  (5)  Other  (specify)  _ 

_  (6)  Other  (specify)  _ 


i.  What  is  the  approximate  total  number  of  semester  hours  of  graduate  credit 
that  you  have  accumulated  in  education? 


j.  What  is  the  approximate  total  number  of  semester  hours  of  undergraduate 
credit  that  you  have  accumulated  in  education? 
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If  applicable,  indicate  approximately  how  many  hours  of  study  you  have 
accumulated  in  the  following  areas? 


k.  Instructional  and 
curriculum  design 

l.  Education  psychology  - 

m.  Tests  and  Measurement  - 

n.  Instructional  systems  design 
or  NAVEDTRA  106A/110 

o.  Evaluation  research  - 

p.  Educational  or  training 
administration  - 

q.  Teaching/instructional 
methods  or  practicum  - 

r.  Other  (specify  -  _ 


University  Military 

level  semester  classroom 

hours _ contact  hours 


) 


If  you  have  served,  or  are  serving,  in  the  military,  approximately  how  many 
military  classroom  contact  hours  have  you  accumulated  in  the  following  areas? 


s.  IT  school  - 


t;  Management  Seminars  -  _  _  _ 

u.  Training  Seminars  - _ _  _ 

v.  Please  list  any  other  formal  or  informal  training  or  experience  you  have 
acquired  that  you  feel  is  relevant  to  your  current  assignment. 


SECTION  II.  CISO  Resource  Requirements 

If  increases  in  the  following  resources  could  be  made  to  your  CISO,  In  what 
priority  do  you  think  such  increases  should  be  made?  Place  a  1  by  the  most 
important,  a  2  by  the  next,  and  so  on. 

a.  _  Number  of  personnel 

b.  _ Skill  and  training  of  personnel 

c.  _ Available  materials 

d.  _  Available  equipment 

e.  _ Available  money 
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For  you  to  perform  effectively  in  your  current  position,  how  useful  would 
short  courses  or  seminars  in  the  following  areas  be? 


NOT  AT 


ALL  USEFUL 

g. 

Curriculum  Writing 

(1) 

h. 

Course  Evaluation 

(1) 

i. 

Instructor  Evaluation 

(1) 

j. 

Instructional  Design 

(1) 

k. 

Test  Development 

(1) 

1. 

Interview  Techniques 

(1) 

m. 

Task  Analysis 

0) 

n. 

Instructional 

Techniques 

(1) 

0. 

Other 

(1) 

SLIGHTLY 

USEFUL 

SOMEWHAT 

USEFUL 

VERY 

USEFUL 

EXTREMELY 

USEFUL 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

(2) 

(3) 

(4) 

(5) 

p.  How  many  courses  do  you  service  as  a  CISO  personnel? 


Please  make  any  additional  comments  or  suggestions  in  the  space  below. 


.  During  the  performance  of  your  2.  On  the  average,  how  often  do  you  3.  Please  check  the  responses  that  best  describe  the  nature  or 


Administer  student 
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Ineck  the  response  that  best  describes  B.  On  the  average,  how  often  do  .you  perform  C.  For  the  functions  you  are  involved 
your  involvement  with  each  function  each  function?  with,  check  the  response  that  best 
listed.  describes  the  amount  of  effort  you 

put  in. 


Please  list  any  additional  major 
Curriculum  and  Instruction  Support 
functions  you  perform  and  answer 
the  questions  for  each  additional 
function. 
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APPENDIX  C 

OTHER  SERVICES*  APPROACHES  TO 
ENSURING  TRAINING  QUALITY 

This  appendix  presents  information  concerning  training  quality 
assurance  policies  and  practices  of  the  Army  and  the  Air  Force. 
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OTHER  SERVICES'  APPROACHES  TO 
ENSURING  TRAINING  QUALITY 

During  the  study,  visits  were  made  to  selected  Arny  and  Air  Force  activi¬ 
ties  to  obtain  information  concerning  these  services'  philosophy,  policies, 
and  practices  regarding  quality  assurance  of  training.  Informal  interviews 
were  conducted  with  personnel  in  various  offices  and  departments,  and  copies 
of  printed  materials  bearing  on  relevant  issues  were  obtained.  No  attempt 
was  made  to  acquire  a  complete  understanding  of  these  other  training  systems. 
Rather,  the  intent  was  to  obtain  information  concerning  particular  concepts 
employed  for  quality  assurance  of  training,  the  manner  in  which  they  have 
been  implemented,  and  the  relative  degree  of  success  of  the  concepts  and 
their  implementation. 

A  variety  of  information  was  obtained  concerning  the  performance  of 
functions  comparable  to  those  expected  of  Navy  CISOs.  This  information  was 
used  as  input  to  both  a  comparative  evaluation  of  Navy  CISOs  and  to  the 
oevelopment  of  recommendations  for  the  future  Navy  performance  of  operations 
to  enhance  training  quality.  Relevant  features  of  Army  and  Air  Force  training 
cuality  assurance  considerations  are  described  below. 

ARMY 


Information  about  Army  quality  assurance  concepts  was  obtained  through 
visits  to  the  following  activities: 

•  Hqs,  Army  Training  and  Doctrine  Command,  Ft.  Monroe,  VA 

•  Army  Training  Management  Institute,  Ft.  Eustis,  VA 

•  Army  Transportation  School,  Ft.  Eustis,  VA 

•  Army  Infantry  School,  Ft.  Benning,  GA 

•  Army  Aviation  Center,  Ft.  Rucker,  AL. 

Army  concern  for  training  quality  is  reflected  in  the  organization  of 
Army  schools.  Information  concerning  this  school  organization  is  presented 
below.  Also  discussed  are  policy  considerations  that  affect  aspects  of 
Army  training  quality  assurance  practices. 

SCHOOL  ORGANIZATION.  Quality  assurance  is  an  integral  consideration  of  the 
Army  training  system.  This  is  reflected  in  the  model  school  organization 
established  by  the  Army's  Training  and  Doctrine  Command.  The  typical  Army 
school  has  four  Type  Directorates.  Each  Directorate,  generally  headed  by 
an  0-6  but  occasionally  an  0-5,  reports  directly  to  an  (the)  assistant  com¬ 
mandant  at  a  school.  Subject  matter  experts  are  typically  assigned  to  each 
Directorate.  Even  though  there  is  a  prescribed  model,  local  commandants  do 
have  latitude  in  deciding  how  best  to  perform  functions  assigned,  and 
organizational  variations  are  also  permitted. 
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The  functions  assigned  three  of  the  Directorates  are  based  on  the  first 
five  phases  of  the  Interservice  Review  Organization  (ITRO)  Instructional 
Systems  Development  (ISO)  Model  as  shown  below: 

DIRECTORATE  ISP  PHASE 

•  Training  Developments  (DTD)  Analyze,  Design,  Develop 

•  Training  and  Doctrine  (DOTD)  Implement 

•  Evaluation  and  Standardization  Evaluate 

(DES) 

The  duties,  responsibilities,  and  practices  of  these  three  Directorates 
are  discussed  below.  The  fourth  Type  Directorate,  Combat  Developments,  is 
also  discussed. 


Directorate  of  Training  Developments.  Directorates  of  Training  Developments 


)  are  generally  charged  with  developing  training  programs  and  products. 


Through  completion  of  the  analysis,  design,  and  development  phases  of  ISD, 
DTDs  are  specifically  to  develop  resident,  nonresident,  and  collective  (unit) 


training. 


Local  DTDs  conduct  task  analytic  efforts  to  define  job  performance 
requirements  for  those  Army  specialties  under  the  cognizance  of  a  given 
school.  In  the  process  of  defining  job  performance  requirements,  local 
DTDs  provide  inputs  to  the  Soldier's  Manual  and  to  the  Commander's  Manual. 
These  inputs  consist  of  statements  of  the  tasks  requiring  performance  for  a 
given  specialty  plus  conditions  and  standards  for  each  level  of  a  Military 
Occupational  Specialty  (MOS).  The  Soldier's  Manual  is  an  important  element 
in  training  quality  control.  It  guides  both  training  development  and  evalua¬ 
tion.  The  Commander's  Manual,  which  is  used  by  field  commanders,  also  lists 
the  tasks  to  be  performed  at  various  MOS  levels.  Further,  it  provides  infor¬ 
mation  concerning  where  and  how  to  train  these  tasks.  For  example,  the 
manual  identifies  correspondence  or  Training  Extension  Courses  (TEC)  that 
are  available  for  particular  training  purposes. 


Although  a  local  0T0  has  nominal  responsibility  for  all  course  develop¬ 
ment,  there  is  variation  in  practice  across  the  schools.  At  some  schools, 
the  training  departments,  not  DTD,  do  necessary  development  work  for  existing 
courses.  DTD  does  the  development  work  only  for  new  courses  (e.g.,  courses 
required  as  a  result  of  new  equipment,  changed  tactics,  or  new  doctrines). 
However,  at  other  schools,  DTDs  may  also  be  involved  in  the  revision  of 
existing  courses. 

The  usual  output  of  a  DTD  is  a  Program  of  Instruction  (POI)  which  typi¬ 
cally  lists  the  recommended  topics  to  be  covered  in  training  and  the  amount 
of  time  to  be  devoted  to  each.  In  some  instances,  PQIs  may  also  identify 
resource  requirements  and  recommend  where  training  should  be  conducted  (i.e., 
institutional  or  unit  training).  Although  a  DTD  may  have  primary  responsi¬ 
bility  for  POIs  for  new  courses,  they  are  generally  coordinated  with  or 
developed  in  conjunction  with  training  departments.  This  may  reflect  a 
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reported  (by  local  center  staff)  shortage  of  skilled  manpower  with  a  trend 
towards  concentrating  SMEs  in  the  instructional  departments. 

For  nonresident  training,  local  DTDs  produce  correspondence  courses 
and  exportable  training  packages  in  areas  for  which  a  particular  school  is 
proponent.  For  example,  the  Army  Aviation  Center  at  Fort  Rucker  (AL)  produces 
courses  related  to  aviation  (e.g.,  air  traffic  control),  the  Transportation 
School  at  Fort  Eustis  (VA)  produces  those  related  to  auto  mechanics.  These 
courses  are  then  taught  at  other  locations  where  such  training  is  needed. 
Currently,  these  exportable  training  packages  consist  principally  of  a  POI 
(the  contents  of  a  POI  were  defined  above)  and  do  not  necessarily  contain 
instructor  or  student  guides  or  testing  materials,  or  instructional  media. 
Generally,  local  OOTs  develop  the  actual  course  including  materials  and 
necessary  tests.  To  the  extent  that  DTDs  do  not  have  their  own  assigned 
SMEs,  development  is  again  shared  with  training  departments. 

Directorate  of  Training  and  Doctrine.  The  Directorate  of  Training  and  Doctrine 
(DOTD)  is  the  instructional  arm  of  an  Army  school.  In  terms  of  the  ISD 
model,  the  Directorate  of  Training  implements  the  instructional  system.  As 
a  result  of  a  recent  TRADOC  decision  (3  Oct  1980)  the  Directorate  of  Training 
also  became  responsible  for  establishing  "Doctrine."  Doctrine  refers  to 
how  particular  tasks  shall  be  executed  (e.g.,  Standard  Operating  Procedures 
for  motor  vehicles). 

In  actual  practice,  considerable  curriculum  development  work  is  performed 
within  the  DOTDs.  This  appears  to  be  primarily  for  existing  courses  and  it 
reflects  the  current  TRAOOC's  view  that  DOTD  SMEs  must  have  a  greater  hand 
in  curriculum  development  than  they  have  had  in  the  past.  (Reasons  for 
this  view  are  discussed  subsequently  in  this  section.) 

Just  as  there  are  variations  from  the  school  model  in  the  functions 
performed  by  Directorates,  there  are  also  allowed  variations  in  the  organi¬ 
zational  structures  at  Army  schools.  For  example,  at  the  U.S.  Army  Infantry 
School  at  Fort  Benning,  Georgia,  the  training  departments  report  to  a  Deputy 
Assistant  Commandant.  The  DOTD  has  the  specialized  function  of  faculty 
development.  The  Armor  School  at  Fort  Knox,  Kentucky,  does  not  have  a  DOTD. 

The  training  departments  report  to  the  Director  for  School  Activities.  At 
Fort  Rucker,  the  training  departments  do  report  to  DOTD  as  suggested  by  the 
school  model. 

Based  on  a  limited  number  of  visits  to  Army  activities,  it  appears 
that  in  the  usual  case  the  DOTDs  do  all  necessary  developmental  work  for 
existing  courses  with  minimal,  if  any,  assistance  from  the  DTDs.  The  DOTDs 
also  evaluate  their  own  instructors  and  exercise  primary  responsibility  for 
test  development,  course  evaluation,  and  student  critique  programs. 

As  mentioned  previously,  proponent  schools  (the  proponent  school  concept 
is  similar  to  the  Navy's  Course  Curriculum  Model  Manager  Concept)  develop 
courses  in  areas  for  which  they  are  responsible.  These  training  packages 
are  then  sent  to  other  locations  where  there  is  a  need  to  teach  particular 
subjects  (e.g.,  automobile  mechanics).  According  to  some  recipients  of 
such  courses,  the  usual  case  is  that  the  proponent  school  supplies  only 
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broad  objectives  with  hours  to  be  spent  on  each  (i.e.,  a  POI).  The  actual 
course  and  course  materials  are  developed  at  the  local  level  by  the  concerned 
DOTD  where  due  consideration  is  given  to  local  factors  that  affect  the  type 
and  design  of  the  course  that  can  be  given.  The  local  training  departments 
usually  also  develop  any  tests  that  will  be  used  with  such  courses. 

Directorate  of  Evaluation  and  Standardization.  The  Directorate  of  Evaluation 
and  Standardization  (DES)  at  a  local  school  is  responsible  for  internal  and 
external  evaluation.  The  responsibility  for  standardization  has  recently 
been  added. 

According  to  TRADOC  sources,  internal  evaluation,  or  quality  assurance, 
essentially  consists  of  determining  that  the  local  DOTD  has  a  training  program 
and  that  the  program  is  implemented.  Thus,  internal  evaluation  is  primarily 
an  audit  function.  External  evaluation  refers  to  the  determination  of  the 
extent  to  which  the  school  meets  the  needs  of  units  in  the  field.  External 
evaluation  involves  the  assessment  of  school  products  (e.g.,  unit  training 
exercises)  and  school  graduates. 

Local  schools  may  send  questionnaires  to  field  supervisors  of  school 
graduates  to  determine  training  adequacy  and  to  highlight  weaknesses  in 
training  or  student  readiness.  At  all  schools,  branch  training  teams  conduct 
external  evaluations.  These  teams,  headed  by  the  local  DES,  may  be  composed 
of  personnel  from  all  Directorates.  The  branch  training  teams  periodically 
visit  field  units  that  receive  students  f>'om  the  courses  (or  products)  for 
which  the  school  is  proponent.  These  visits  have  two  purposes:  to  inform 
field  units  of  what  the  school  does  and  to  determine  any  inadequacies  exper¬ 
ienced  in  the  field. 

Practices  at  field  units  appear  to  reflect  quite  closely  the  notions 
described  above.  The  Evaluation  Division  of  DES  at  the  USA  Aviation  Center, 
for  example,  has  an  internal  evaluation  branch  and  an  external  evaluation 
branch.  The  internal  evaluation  branch  is  concerned  with  evaluation  of 
Instructional  Systems  Development  practices  within  the  Aviation  Center. 

This  branch,  for  selected  courses,  has  examined  the  audit  trail  through  the 
first  3  phases  of  ISO  to  ensure  that  ISD  procedures  were  followed.  Currently, 
the  branch  does  "snapshot"  evaluations;  e.g.,  "visiting"  courses,  looking 
at  documentation  to  see  if  objectives  are  consistent,  effective. 

The  External  Evaluation  Branch  is  the  coordinator  for  the  Army  Aviation 
Center  Training  Analysis  and  Assistance  Team  (AACTAT).  This  team  visits 
Army  installations  worldwide  to  interview  pilots  and  aviation  enlisted  personnel. 
They  evaluate  courses  for  which  the  Aviation  Center  is  proponent.  The  External 
Evaluation  Branch  also  routinely  follows  up  (quarterly)  graduates  of  five 
different  courses  taught  at  the  center.  A  questionnaire  based  on  the  Job 
Task  Inventory  (JTI)  from  the  course  development  effort  is  used  for  this. 

Other  courses  are  done  on  an  as-needed  basis.  No  effort  is  expended  to 
evaluate  training  extension  courses  (TEC)  or  correspondence  courses,  however. 

It  is  understood  that  DESs  at  other  Army  schools  function  in  quite 
similar  ways  to  the  Aviation  Center  DES.  Fort  Knox,  however,  does  not  have 
an  equivalent  to  DES.  Evaluation  and  standardization  functions  are  performed 
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by  the  Office  of  Armor  Force  Management.  The  major  emphasis  of  evaluation 
at  all  schools  is  product  evaluation  accomplished  through  visits  to  operating 
units  and/or  questionnaires  to  graduates  and  field  commanders.  Evaluation 
results  are  provided  to  the  training  departments  for  their  use  and  final 
disposition.  DES  Branch  teams,  in  addition  to  checking  on  graduate  quality 
at  field  sites,  also  examine  the  appropriateness  of  instructional  materials 
prepared  by  the  schools  for  field  use. 

Directorate  of  Combat  Developments.  The  fourth  Type  Directorate,  the  Director- 
ate  of  Combat  Developments  (DCD),  has  primary  responsibility  at  a  school 
for  assuring  the  integration  of  new  hardware  into  field  units.  This  director¬ 
ate  provides  inputs  to  the  other  directorates  who  prepare,  conduct,  and 
evaluate  required  training. 

POLICY  CONSIDERATIONS.  The  Army  Training  and  Doctrine  Command  (TRADOC) 
sets  policies  for  implementation  at  the  school  level.  Each  of  the  divisions 
in  a  school  has  a  counterpart  at  TRADOC.  Thus,  the  Training  Development 
Institute  sets  policies  for  the  development  of  training  programs  at  schools; 
the  Training  Mar'£>3ement  Institute  (TMI)  sets  policies  for  implementation  by 
Directorates  of  Training  in  schools;  and  the  Evaluation  Directorate  at  TRADOC 
sets  policies  fo;  ‘valuation  at  the  school  level.  Currently,  the  Evaluation 
Directorate  performs  primarily  an  audit  function  aimed  at  determining  that 
local  schools  follow  prescribed  steps  in  the  training  development  process. 

Current  indications  are  that  several  policy  changes  are  imminent  which 
will  affect  local  schools'  operations.  Ouring  discussions  with  various 
Army  personnel,  at  TRADOC  and  at  local  schools,  several  predictions  and 
recommendations  for  changes  in  policy  were  repeated.  There  seems  to  be  a 
general  consensus  that  training  development  and  implementation  should  be 
left  to  the  discretion  of  the  local  school  commandant.  The  emphasis  in 
policy  should  be  on  the  desired  outcomes  for  a  particular  training  program 
and  on  accountability  for  failure  to  provide  training  that  results  in  those 
desired  outcomes.  Additionally,  performance  evaluation  should  occur  prior 
to  designing  a  training  program  to  determine  if  the  program  is  indeed 
necessary. 

A  consensus  was  expressed  at  Command  levels  that  there  is  room  for 
improvement  in  standardizing  Army  training.  However,  the  current  emphasis 
in  standardization  is  on  the  process  of  training,  not  the  outcome.  This  is 
contradictory  with  the  philosophy  of  leaving  as  much  control  as  possible  to 
the  local  commandant.  Suggestions  concerning  standardization  converged  on 
the  concept  of  standardizing  only  the  outcomes  of  training  programs.  Attempts 
to  standardize  the  actual  training  process  should  be  minimized,  and  the 
local  commandant  should  have  latitude  to  tailor  the  training  program  to 
suit  local  conditions. 

In  conjunction  with  a  shift  from  standardization  of  process  to  standardiza¬ 
tion  of  outcomes,  an  increase  in  the  accountability  of  the  local  commandant 
would  be  necessary.  The  local  commardant  would,  in  effect,  be  held  responsible 
for  ensuring  that  the  products  of  his  school  were  indeed  satisfactory.  For 
such  accountability  to  be  successful,  there  would  have  to  be  visible  sanctions 
from  higher  authority  if  acceptable  standards  were  not  met. 
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It  appears  that  training  development  functions  are  being  shifted  to 
the  local  DOTD  level.  Several  reasons  underlie  this  policy  change.  It  is 
believed  that  the  local  instructors  and  SME's  (i.e.,  DOT  personnel)  should 
have  significant  input  into  the  curriculum  development  process  since  they 
know  the  most  about  the  performance  of  a  particular  job.  Further,  instructors 
who  participate  in  curriculum  development  will  feel  a  sense  of  authorship 
and  will  be  more  willing  to  devote  the  necessary  time  and  energy  to  make 
the  training  effective.  If  local  instructors  are  not  included,  they  will 
modify  the  curriculum  anyway  since  they  are  the  ones  who  must  use  it.  Includ¬ 
ing  them  in  development  at  the  start  will  result  in  a  better  curriculum  due 
to  the  pooling  of  their  experience  and  expertise.  Finally,  the  local  instruc¬ 
tors  are  more  aware  of  local  conditions  than  the  proponent  schools  may  be. 

AIR  FORCE 

Information  about  Air  Force  training  quality  assurance  policies  and 
practices  was  obtained  from  visits  to  the  following  Air  Force  activities: 

•  Air  Training  Command,  Randolph  AFB,  TX 

•  Standards  and  Evaluation  Directorate,  Randolph  AFB,  TX 

•  Occupational  Measurement  Center,  Randolph  AFB,  TX 

•  3200th  Technical  Training  Wing,  Lackland  AFB,  TX 

•  3300th  Technical  Training  Wing,  Keesler  AFB,  MS 

Information  concerning  Air  Force  performance  of  functions  comparable 
to  those  expected  of  Navy  CISOs  is  presented  below.  The  Air  Force  school 
organization  is  presented  first.  This  is  followed  by  brief  descriptions  of 
key  features  of  Air  Force  practices  that  are  relevant  to  developing  recom¬ 
mendations  for  Navy  approaches  to  ensuring  training  quality. 

SCHOOL  ORGANIZATION.  Air  Force  training  is  highly  decentralized.  The  Air 
Training  Command  { ATC)  establishes  policies  and  monitors  the  schools  for 
compliance.  However,  since  1977  responsibility  for  the  quality  of 
particular  training  courses  has  been  vested  at  the  local  school  level. 

Air  Force  technical  training  is  conducted  under  the  aegis  of  Technical 
Training  Wings  located  at  various  Air  Force  bases.  Figure  C-l  displays 
schematically  selected  elements  of  a  typical  Training  Wing  that  are 
directly  concerned  with  training.  The  Wing  Commander  reports  to  the  Air 
Force  Base  Commander,  who  in  turn  reports  to  the  ATC.  Several  specialized 
staff  functions  are  placed  at  Wing  level.  Actual  training  is  conducted 
within  Technical  Training  Groups  under  the  Wing.  Duties  and  functions  of 
selected  organizational  elements  are  briefly  described  below. 

Wing  Staff  Functions.  Relevant  Wing  staff  functions  are  listed  below. 
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Figure  C-1.  Typical  Air  Force  Technical  Training  Wing  Organization 
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Training  Advisor.  The  Training  Advisor  at  the  Wing  level  Is  the  senior 
civilian  training  specialist  (GS  1712  series--the  GS  1710  series  Is  rarely 
used  by  the  Air  Force)  within  the  Wing.  In  addition  to  advising  the  Wing 
Commander  about  training  matters,  the  training  advisor  serves  as  a  direct 
liaison  to  various  ATC  activities  and  to  major  command  users  of  graduates 
concerning  changing  user  needs  and  problems. 

Plans  and  Requirements  Division.  Career  field  training  managers  are  located 
within  the  Plans  and  Requirements  Division  of  a  Wing.  These  individuals 
set  the  pattern  for  all  training  (formal  institutional  training,  correspond¬ 
ence  courses,  and  on-the-job  (OJT)  training)  required  to  establish  appropriate 
skills  and  knowledges  across  all  levels  of  an  Air  Force  job  specialty.  The 
Air  Force  emphasis  is  on  planning/training  for  a  total  career  field  rather 
than  on  single  courses  of  instruction  given  in  isolation  from  the  total 
career  field. 

Training  Managers  within  a  Wing's  Plans  and  Requirements  Division  are 
typically  GS-12  training  specialists.  They  are  directly  responsible  to  the 
Air  Force  for  particular  courses  of  instruction  given  under  the  Wing.  They 
exercise  administrative  and  technical  control  over  course(s).  Training 
managers  function  in  ways  similar  to  Navy  Training  Program  Coordinators. 
However,  they  have  far  greater  authority  and  are  located  at  the  training 
activity  level  rather  than  within  a  headquarters  group.  Twenty  Training 
Managers  are  placed  at  Keesler  Air  Force  Base--one  for  each  Air  Force  Speci¬ 
alty  (AFS)  served  by  this  base. 

Faculty  Development  Division.  The  Faculty  Development  Division  at  the  Wing 
provides  all  training,  both  preservice  and  inservice,  for  new  instructors 
reporting  to  the  Wing.  The  model  for  instructor  courses  is  designed  at 
Sheppard  Air  Force  Base.  All  enlisted  instructors  are  required  to  receive 
36  hours  of  inservice  training  annually.  This  may  be  given  by  the  Faculty 
Development  Division  or  within  the  particular  training  branch  to  which  the 
instructor  is  assigned.  All  courses  are  required  in  order  for  an  individual 
to  receive  the  Master  Instructor  Certificate. 

Training  Evaluation  Division.  The  Training  Evaluation  Division  conducts 
external  evaluations  (via  questionnaires  or  field  visits)  of  courses  for 
which  the  Wing  is  responsible.  The  division  also  assists  the  training  branches 
in  internal  evaluation  efforts  (e.g.,  by  preparing  checklists). 

Training  Services  Division.  The  Training  Services  Division  produces  training 
aids  and  provides  word  processing  and  other  support  to  the  training 
mission. 

Office  of  Foreign  Military  Affairs.  Special  administrative  and  counseling 
support  for  foreign  military  students  is  provided  by  the  Foreign  Military 
Affairs  Office. 

Student  Affairs  Division.  The  Student  Affairs  Division  contains  student 
advisors  and  counselors.  Counseling  is  given  for  both  academic  and 
nonacademic  problems.  However,  particular  academic  problems  are  referred 
to  appropriate  instructional  personnel  in  the  Training  Branches. 
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Technical  Training  Groups.  Typically,  four  to  six  Technical  Training  Groups 
(TTG)  are  organized  under  the  Training  Wing.  Technical  training  is  accomplished 
by  Training  Branches  within  these  Groups. 

Training  Branches.  As  shown  in  figure  C-l,  each  TTG  is  composed  typically 
of  three  Training  Branches.  In  turn,  the  branches  typically  consist  of  a 
Curriculum  Development  Section  (CDS)  and  three  Course  Areas.  The  CDS  is 
specifically  responsible  for  the  development  of  curricula  and  instructional 
materials  (e.g.,  student  guides,  instructor  guides,  tests,  training  aids) 
that  will  be  used  for  the  courses  taught  by  the  Branch.  Instructors 
assigned  to  the  Course  Areas  conduct  the  instruction  in  accordance  with  the 
CDS  plans. 

KEY  FEATURES  OF  AIR  FORCE  TRAINING 

Selected  features  of  Air  Force  training  of  direct  interest  for 
training  quality  assurance  are  described  below. 

SPECIALTY  TRAINING  STANDARD.  The  Spe:ialty  Training  Standard  (STS)  is  a 
key  document  which  is  used  by  the  Air  Force  as  a  firm  basis  for  all 
training  (schoolhouse,  correspondence  courses,  and  OJT)  and  testing  for  a 
given  Air  Force  Specialty  Code  (AFSC). 

An  STS  lists  skills  and  knowledges  needed  for  each  level  of  an  AFSC. 

It  also  specifies  the  level  of  knowledge  and/or  the  level  of  performance 
proficiency  required  at  each  step  of  the  AFSC.  Thus,  the  STS  entries 
identify  specifically  what  is  to  be  accomplished  by  training  (and  also 
define  legitimate  testing/evaluation  items). 

An  STS  may  be  initially  compiled  by  the  Training  Manager  who  is 
concerned  with  a  given  AFSC.  However,  it  is  validated  by  surveys  conducted 
by  the  Air  Force  Occupational  Measurement  Center  (OMC).  The  OMC  is  a 
staff  function  of  Headquarters  ATC.  The  OMC  surveys  each  Air  Force 
specialty  approximately  every  4  years.  In  addition  to  other  information, 
these  surveys  collect  data  from  AFSC  incumbents  concerning  their 
backgrounds,  tasks  performed,  the  amount  of  time  spent  on  each  task,  and 
the  frequency  of  task  performance.  The  survey  data  is  used  for  a  variety 
of  purposes.  A  principal  use  is  to  update  the  STS  for  the  AFSC. 

Training  Managers  at  the  schools  may  convene  a  Utilization  and 
Training  Workshop  (UTW)  to  review  survey  data.  The  UTW  is  composed  of 
representatives  from  the  ATC  and  all  najor  user  commands.  The  UTW  reviews 
the  survey  data  to  assess  discrepancies  between  what  is  currently  being 
trained  and  what  jobs  incumbents  report  they  currently  do.  Difficulty  of 
task  performance  is  evaluated  by  the  Subject  Matter  Specialists  (SMS)  at 
the  UTW  and  decisions  are  made  from  tie  data  about  future  training. 

Information  concerning  the  number  of  respondents  performing  a  particular 
task(s)  is  used  in  the  following  way: 

•  less  than  30  percent  performing  a  given  task— no  school  training 
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•  30-50  percent  performing  a  task--knowledge  training  only  at  the 
school 

•  50  percent  or  more  performing  a  task--hands-on  performance  training 
at  the  school. 

Job  tasks  revealed  by  the  survey  to  be  required  only  for  one  major 
command  are  not  routinely  selected  for  formal  training.  Training  for  these 
tasks  becomes  the  responsibility  of  that  command.  One  product  of  the  UTW 
is  an  updated  STS  for  the  AFSC  of  concern.  The  STS  provides  a  basis  for  a 
contractual  relationship  between  the  school  and  the  major  user  commands. 

It  specifies  what  the  school  will  train  and  to  what  level.  It  is  the  respon¬ 
sibility  of  the  local  Wing  to  ensure  that  school  graduates  meet  the  criteria 
specified  in  the  STS. 

Survey  data,  used  in  updating  the  STS,  may  signal  the  need  for  revisions 
to  current  training.  The  need  for  revisions  to  courses  may  also  be  signaled 
in  other  ways.  For  example,  the  Training  Evaluation  Division  at  the  local 
Wing  conducts  periodic  external  training  evaluations.  These  evaluations 
focus  on  specific  items  listed  on  an  STS.  To  the  extent  that  these  evalua¬ 
tions  determine  that  a  school  is  not  meeting  its  contractual  obligations  to 
the  major  commands,  training  must  be  adjusted.  (Another  possible  outcome 
of  such  evaluations,  however,  may  be  a  request  for  a  new  occupational 
survey  to  determine  the  STS  validity.) 

CURRICULUM  DEVELOPMENT.  The  majority  of  required  course  development  work 
occurs  at  the  local  Training  Wing.  The  local  wing  is  totally  responsible 
for  revisions  of  old  courses.  Course  development  for  new  weapons  systems 
is  the  responsibility  of  the  Air  Training  Command.  The  3306th  Test  and 
Evaluation  (T  and  E)  Squadron  (Edwards  AFB,  California)  is  specifically 
responsible  within  ATC  for  these  developments.  Subject  Matter  Specialists 
are  recruited  as  needed  from  all  over  the  Air  Force,  given  a  general  course 
on  ISO  (the  Air  Force  uses  its  own  version  of  ISD  as  contained  in  Air  Force 
Manual  50-2).  They  then  assist  in  developing  the  required  new  course(s). 

The  T  and  E  Squadron,  as  a  minimum,  determines  how  the  new  weapons  system 
is  (to  be)  operated  and  maintained  and  establishes  training  requirements. 

This  material  may  then  be  handed  over  to  the  appropriate  Wing(s)  for 
further,  more  detailed  course  development  work.  Certain  new  non-weapons 
system  course  development  efforts  may  also  be  handled  by  ATC.  Courses  in 
this  category  remain  under  ATC  control  until  it  has  been  determined  that 
they  are  functioning  smoothly.  Subsequently,  they  also  are  handed  over  to 
an  appropriate  Training  Wing  for  maintenance. 

Curriculum  development  at  the  Wing  level  is  a  multistage  process. 
Initially,  the  Training  Manager  develops  a  Course  Training  Standard  (CTS). 

The  CTS,  based  on  the  STS  for  the  AFSC,  specifies  the  training  to  be  given 
and  the  levels  to  which  students  will  be  trained.  The  Training  Manager 
also  develops  a  Course  Chart  showing  event  sequencing  and  course 
scheduling.  These  two  items  represent  the  basic  control  documentation  for 
a  course  and  they  also  are  viewed  as  contractual  (with  user  commands) 
items. 
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The  CTS  and  Course  Chart  are  subsequently  given  to  the  Curriculum 
Development  Section  (CDS)  in  the  appropriate  Group  Training  Branch.  CDS 
then  develops  the  course.  Most  often,  instructors  will  be  borrowed  from  a 
Course  Area  to  assist  in  this  effort.  A  GS-11  training  specialist  (or  GS-9 
temporarily  promoted  to  GS-11)  typically  leads  the  ISD  development  team. 

CDS  produces  a  detailed  Plan  of  Instruction  (POI)  which  states  the  learning 
objectives,  student  instructional  materials,  audio-visuals  to  be  used,  train¬ 
ing  methods,  instructional  guidance  to  be  used,  etc.  Tests  to  be  used  for 
the  course  are  also  developed  by  CDS.  Currently,  there  is  no  Air  Force¬ 
wide  requirement  for  comprehensive  end-of -course  testing.  Testing  usually 
occurs  after  each  block  (module)  o'  instruction.  The  general  policy  on 
student  testing  is  that  every  objective  in  the  POI  will  be  tested. 

From  the  Branch  CDS,  the  POI  goes  to  the  Course  Area.  Each  instructor 
is  then  required  to  prepare  his  owi  individual  lesson  plan(s)  specifying 
how  he  will  conduct  the  training.  These  lesson  plans  may  be  reviewed  by 
CDS,  but  most  often  these  reviews  ire  performed  by  the  Chief  Instructor. 

Air  Force  general  policy  is  t>  develop  courses  for  group-paced  instruc¬ 
tion.  According  to  ATC  sources,  oily  about  10  percent  of  Air  Force  instruc¬ 
tion  occurs  under  self-paced  methois,  and  these  are  mostly  advanced  courses. 

A  prevalent  belief  is  that  self-pa :ed  instruction  is  not  as  effective  as 
group-paced  instruction. 

Minor  revisions  to  courses  which  involve  no  changes  to  time  or  other 
resources  can  be  approved  by  the  Training  Manager  at  the  local  Wing  level. 
Revisions  that  require  additional  capital  resources,  greater  time,  or  increased 
manpower  can  be  approved  by  the  local  Wing  Commander.  However,  to  the  extent 
that  the  necessary  resources  are  not  immediately  available  to  implement  the 
revisions,  they  must  be  solicited  From  the  ATC. 

EVALUATION  FUNCTIONS 

Most  often,  local  Training  Branches  are  not  directly  involved  in  collect¬ 
ing  external  evaluation  information.  This  information  is  obtained  in  a 
variety  of  ways  and  is  provided  to  the  local  Training  Branches.  Some  external 
evaluation  sources  have  been  previously  mentioned;  greater  detail  on  Air 
Force  external  evaluation  programs  is  provided  in  Hall,  et  al.  (1976). 

Internal  evaluation  at  Tocal  Air  Force  levels  involves  several  functions 
similar  to  those  expected  of  Navy  CISOs;  e.g.,  instructor  evaluation,  annual 
course  reviews,  student  critiques.  Instructor  evaluations  are  conducted  by 
instructor  supervisors.  In  addition  to  assessing  instructors'  presentation 
of  material,  continuous  staff  surveillance  is  maintained  to  ensure  that  the 
individual  instructor  follows  the  POI.  Annual  Quality  Reviews  ( AQR)  are 
also  conducted  for  each  course.  These  reviews  involve  a  self-audit  of  equip¬ 
ment,  materials,  and  instructors.  AQRs  are  done  by  the  particular  Course 
Area  under  the  cognizance  of  the  Branch  CDS.  The  AQR  format  is  prepared  by 
the  Wing  Training  Evaluation  Division.  The  CDS  schedules  the  review,  ensures 
that  the  schedule  is  adhered  to,  and  that  all  aspects  of  the  course  are 
examined.  At  least  two  individuals  at  the  course  level  must  sign  that  the 
AQR  has  been  accomplished. 
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Each  Course  Area  also  administers  a  student  critique  program.  This  is 
handled  by  the  training  instructors.  Students  are  provided  opportunity  to 
comment  on  the  quality  of  the  instruction  they  received.  Periodically, 
student  critiques  are  reviewed  by  higher  authority  at  the  Wing. 

Test  item  analysis  is  accomplished  within  the  Course  Area.  Each  time, 
a  test  is  given,  the  individual  instructor  must  complete  a  test  item  analysis 
form  which  is  reviewed  by  the  instructor  supervisor.  These  forms  are  then 
used  to  determine  which  items  "are  not  working  right."  The  instructor  super¬ 
visor  can  recommend  changes  to  tests  at  any  time.  Once  per  quarter,  the 
Branch  CDS  reviews  the  test  item  analysis  forms,  and  the  supervisor's  comments 
about  particular  items.  As  necessary,  test  items  .ire  rewritten  by  the  course 
instructors  witn  assistance  from  CDS. 
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APPENDIX  D 

FACTOR  AND  DISCRIMINANT  FUNCTION  ANALYSES  OF  CISO  FUNCTIONS 

This  appendix  presents  the  technical  details  of  the  factor  and 
discriminant  function  analyses  performed  on  data  from  the  long  question¬ 
naire.  The  details  are  relevant  to  the  types  of  functions  CISOs  perform 
and  to  the  determination  of  different  types  of  CISOs. 
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FACTOR  AND  DISCRIMINANT  FUNCTION  ANALYSES  OF  CISO  FUNCTIONS 
TYPES  OF  FUNCTIONS 

The  first  step  in  determining  a  typology  of  CISOs  was  to  examine  the 
functions  performed  to  find  an  underlying  structure  that  would  allow  for 
the  reduction  of  separate  functions  into  groups.  Principle  components 
analysis  without  iterations  was  chosen  because  it  allows  for  the  presenta¬ 
tion  of  single  items  or  variables  in  components  that  are  mathematically 
equivalent  to  the  original  items  (Nunnally,  1978).  The  items  used  in  the 
principle  components  analysis  were  from  section  IV  of  the  long  form.  In 
subsection  I,  items  a  through  i  for  question  A  were  chosen;  in  subsection 
II,  items  a  through  n  for  question  A  were  chosen  (see  appendix  B).  The 
five  factors  with  eigenvalues  greater  than  or  equal  to  1.0  were  retained 
for  Varimax  rotation.  The  rotated  5-factor  solution  is  shown  in  table  D-l. 

Based  on  an  examination  of  the  loadings  in  the  rotated  factor 
structure,  there  appear  to  be  five  fairly  well  defined  factors.  Four 
scales  were  created  based  on  the  four  factors  that  have  multiple  high 
loadings.  The  first  scale  consists  of  the  mean  of  items  a  through  h  in 
subsection  II  and  is  called  EVAL1.  The  second  scale  consists  of  the  mean 
of  a  through  e  in  subsection  I  and  is  called  CISWRK.  The  third  scale 
consists  of  the  mean  of  items  f,  g,  h,  k,  and  m  in  subsection  I  and  is 
called  CISOEV.  The  fourth  scale  consists  of  the  mean  of  items  j  and  1  in 
subsection  II  and  is  called  EVAL2.  The  fifth  factor  only  has  one  strong 
loading  from  item  a  in  subsection  II.  This  item  is  used  as  a  single  item 
scale  called  TECLIB.  The  actual  functions  constituting  these  five  scales 
are  shown  in  table  5  in  the  text,  page  31. 

Internal  consistency  reliabilities  were  computed  for  the  four 
multiple-item  scales  using  coefficient  alpha  (Nunnally,  1978).  EVALl  had  a 
reliability  of  0.90;  CISWRK  had  a  reliability  of  0.82;  CISDEV  had  a 
reliability  of  0.75;  and  EVAL2  had  a  reliability  of  0.62. 

Next,  one-way  ANOVA's  were  performed  using  the  five  scales,  CISWRK, 
EVALl,  CISOEV,  EVAL2,  and  TECLIB,  as  dependent  variables  and  school 
identifying  number  as  the  independent  factor.  This  analysis  was  performed 
to  determine  if  the  scales  reflect  characteristics  about  each  CISO  rather 
than  about  the  individual  respondents.  If  the  F-ratio  is  significant,  then 
that  indicates  that  the  scale  measures  a  school -level  characteristic 
(Borgatta  and  Jackson,  1980).  Three  of  the  scales,  CISDEV,  EVAL2,  and 
TECLIB  had  significant  F-ratios  (p<.05).  Thus,  these  three  scales  were 
used  to  determine  if  there  were  different  types  of  CISOs  with  respect  to 
involvement  with  functions. 
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TABLE  0-1.  VARIMAX  ROTATEO  FACTOR  LOADINGS  OF  DEGREE  OF 
INVOLVEMENT  WITH  FUNCTIONS 


1  FACTORS  S 

FUNCTIONS 

EVAL1 

CISWRK 

CISDEV  EVAL2 

TECLIB 

REVIEW  COURSE  AND 

CURRICULA  DATA  AND 
DOCUMENTATION 

.67 

DO  TASK  ANALYSIS 

.64 

DEVELOP  CURRICULA 

.87 

DESIGN/REVISE  COURSES 

AND  CURRICULA 

.86 

DEVELOP  TRAINING 

MATERIALS  AND  AIDS 

.60 

MONITOR  DEVELOPMENTS 

IN  TRAINING  TECHNOLOGY 

.69 

SELECT  INSTRUCTIONAL 
DELIVERY  SYSTEMS 

.73 

DEVELOP/CONDUCT 

INSERVICE  TRAINING 

.60 

MAINTAIN  CENTRAL 

TECHNICAL  LIBRARY 

.89 

’REPARE  EXAMINATIONS 

.73 

ANALYZE  TEST  DATA 

.77 

MAINTAIN  TEST  ITEM  BANK 

.82 

DEVELOP  INTERNAL  FEEDBACK 
INSTRUMENTS 

.65 

DEVELOP  ITEMS  FOR 

EXTERNAL  FEEDBACK 

.55 

ANALYZE  AND  INTERPRET 
:EEDBACK  DATA 

.65 

STUDY  ATTRITION 

.61 
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TABLE  D-l.  VAR I MAX  ROTATEO  FACTOR  LOADINGS  OF  DEGREE  OF 
INVOLVEMENT  WITH  FUNCTIONS  (continued) 


FACTORS 

FUNCTIONS 

EVALl 

CISWRK 

CISDEV 

EVAL2 

STUDY  SETBACKS 

.57 

ADMINISTER  STUDENT 

CRITIQUE  PROGRAM 

.56 

EVALUATE  INSTRUCTORS 

.64 

EVALUATE  INSERVICE 

PROGRAMS 

.43 

CONDUCT  ANNUAL  COURSE 
REVIEWS 

.60 

CONDUCT  SPECIAL 

RESEARCH  PROJECTS 

.52 
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TYPES  OF  CISOS 

In  order  to  determine  if  there  are  different  types  of  CISOs  with 
respect  to  functions  performed,  discriminant  function  analysis  (Nunnally, 
1978)  using  the  three  CISO-level  scales,  CISOEV,  EVAL2,  and  TECLIB,  was 
chosen.  The  school  was  the  dependent  variable.  Three  functions  were 
extracted  and  rotated  using  the  VARIMAX  criterion.  The  first  function,  on 
which  TECLIB  loads  highest.  Is  significant  at  the  .0001  level,  indicating 
that  it  provides  a  great  deal  of  discriminatory  information.  The  second 
function,  representing  primarily  the  scale  EVAL2,  is  also  significant  (p  = 
.0059).  The  third  function,  representing  CISOEV,  is  significant  at  the  .10 
level.  Although  this  is  lower  than  the  traditional  standard  of  .05,  the 
third  function  is  used  for  discrimination  because  it  still  contains  some 
discriminatory  information.  The  29  school  scores  on  the  three  discriminant 
functions  are  presented  in  table  D-2.  Schools  that  have  similar  scores  on 
all  three  functions  can  be  considered  similar  in  terms  of  the  degree  of 
involvement  with  the  functions  represented  in  the  three  scales,  CISDEV, 
EVAL2,  and  TECLIB.  A  visual  inspection  of  the  scores  resulted  in  the  six 
clusters  of  CISOs  shown  in  table  6  in  section  IV. 
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TABLE  D-2.  CANONICAL  SCORES  FOR  29  CISOs  ON 
THREE  DISCRIMINANT  FUNCTIONS 

FUNCTION  1  FUNCTION  2 


FUNCTION  3 


CISO 


TECLIB 


EVAL2 


CISDEV 
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TABLE  0-2.  CANONICAL  SCORES  FOR  29  CISOs  ON 

THREE  DISCRIMINANT  FUNCTIONS  (continued) 


CISO 

FUNCTION  1 

TECLIB 

FUNCTION  2 

EVAL2 

FUNCTION  3 

CISDEV 

NAVTECHTRACEN  Meridian 

-.66 

-.08 

1.63 

FLECOMBATRACEN  San  Diego 

-1.01 

1.11 

.74 

SUBTRAFAC  San  Diego 

-.75 

.76 

-.31 

iAVTECHTRACEN  Treasure  Island 

1.65 

-.49 

.22 

COMBATS YSTECHSCOL  Mare  Island 

-.88 

.48 

.54 

NAVPHIBSCOL  Coronado 

-.44 

.59 

.63 

iAMTRAGRU  Memphis 

-.42 

-.50 

.31 
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